package-manager-3.0.1/0000755000175000017500000000000014565163601012633 5ustar rharhapackage-manager-3.0.1/COPYING0000644000175000017500000000351614565163601013673 0ustar rharhaCopyright (c) 2016 The Board of Trustees of the University of Illinois. All rights reserved. Developed by: Cybersecurity Directorate National Center for Supercomputing Applications University of Illinois http://illinois.edu Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal with the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimers. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimers in the documentation and/or other materials provided with the distribution. * Neither the names of the National Center for Supercomputing Applications, the University of Illinois, nor the names of its contributors may be used to endorse or promote products derived from this Software without specific prior written permission. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE SOFTWARE. Note that some files in the distribution may carry their own copyright notices. package-manager-3.0.1/.update-changes.cfg0000644000175000017500000000065114565163601016264 0ustar rharha# Automatically adapt version in files. function replace_python_package_version { file=$1 version=$2 cat $file | sed "s#^\\( *__version__ *= *\\)\"\\([0-9.-]\\{1,\\}\\)\"#\1\"$version\"#g" >$file.tmp cat $file.tmp >$file rm -f $file.tmp git add $file } function new_version_hook { version=$1 replace_python_package_version zeekpkg/__init__.py $version make man git add ./doc/man } package-manager-3.0.1/doc/0000755000175000017500000000000014565163601013400 5ustar rharhapackage-manager-3.0.1/doc/docutils.conf0000644000175000017500000000003314565163601016071 0ustar rharha[parsers] smart_quotes: no package-manager-3.0.1/doc/zkg.rst0000644000175000017500000000601114565163601014723 0ustar rharha.. _zkg: zkg Command-Line Tool ===================== .. argparse:: :module: zkg :func: argparser :prog: zkg :nosubcommands: --configfile : @after See :ref:`zkg-config-file`. Commands -------- .. _test-command: test ~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: test .. _install-command: install ~~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: install .. _remove-command: remove ~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: remove .. note:: You may also say ``uninstall``. .. _purge-command: purge ~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: purge .. _bundle-command: bundle ~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: bundle .. _unbundle-command: unbundle ~~~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: unbundle .. _refresh-command: refresh ~~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: refresh .. _upgrade-command: upgrade ~~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: upgrade .. _load-command: load ~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: load .. _unload-command: unload ~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: unload .. _pin-command: pin ~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: pin .. _unpin-command: unpin ~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: unpin .. _list-command: list ~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: list .. _search-command: search ~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: search .. _info-command: info ~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: info .. _config-command: config ~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: config .. _autoconfig-command: autoconfig ~~~~~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: autoconfig .. _env-command: env ~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: env create ~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: create template info ~~~~~~~~~~~~~ .. argparse:: :module: zkg :func: argparser :prog: zkg :path: template info .. _zkg-config-file: Config File ----------- The :program:`zkg` command-line tool uses an INI-format config file to allow users to customize their :doc:`Package Sources `, :doc:`Package ` installation paths, Zeek executable/source paths, and other :program:`zkg` options. See the default/example config file below for explanations of the available options and how to customize them: .. literalinclude:: ../zkg.config package-manager-3.0.1/doc/source.rst0000644000175000017500000001003114565163601015425 0ustar rharha.. _Zeek Packages Git Repository: https://github.com/zeek/packages How-To: Create a Package Source =============================== :ref:`zkg `, by default, is configured to obtain packages from a single "package source", the `Zeek Packages Git Repository`_, which is hosted by and loosely curated by the Zeek Team. However, users may :ref:`configure zkg ` to use other package sources: either ones they've set up themselves for organization purposes or those hosted by other third parties. Package Source Setup -------------------- In order to set up such a package source, one simply has to create a git repository and then add :ref:`Package Index Files ` to it. These files may be created at any path in the package source's git repository. E.g. the `Zeek Packages Git Repository`_ organizes package index files hierarchically based on package author names such as :file:`alice/zkg.index` or :file:`bob/zkg.index` where ``alice`` and ``bob`` are usually GitHub usernames or some unique way of identifying the organization/person that maintains Zeek packages. However, a source is free to use a flat organization with a single, top-level :file:`zkg.index`. .. note:: The magic index file name of :file:`zkg.index` is available :program:`since zkg v2.0`. For compatibility purposes, the old index file name of :file:`bro-pkg.index` is also still supported. After creating a git repo for the package source and adding package index files to it, it's ready to be used by :ref:`zkg `. .. _package-index-file: Package Index Files ------------------- Files named :file:`zkg.index` (or the legacy :file:`bro-pkg.index`) are used to describe the :doc:`Zeek Packages ` found within the package source. They are simply a list of git URLs pointing to the git repositories of packages. For example:: https://github.com/zeek/foo https://github.com/zeek/bar https://github.com/zeek/baz Local filesystem paths are also valid if the package source is only meant for your own private usage or testing. Adding Packages --------------- Adding packages is as simple as adding new :ref:`Package Index Files ` or extending existing ones with new URLs and then commiting/pushing those changes to the package source git repository. :ref:`zkg ` will see new packages listed the next time it uses the :ref:`refresh command `. Removing Packages ----------------- Just remove the package's URL from the :ref:`Package Index File ` that it's contained within. After the next time :program:`zkg` uses the :ref:`refresh command `, it will no longer see the now-removed package when viewing package listings via by the :ref:`list command `. Users that had previously installed the now-removed package may continue to use it and receive updates for it. Aggregating Metadata -------------------- The maintainer/operator of a package source may choose to periodically aggregate the metadata contained in its packages' :file:`zkg.meta` (and legacy :file:`bro-pkg.meta`) files. The :ref:`zkg refresh ` is used to perform the task. For example: .. code-block:: console $ zkg refresh --aggregate --push --sources my_source The optional ``--push`` flag is helpful for setting up cron jobs to automatically perform this task periodically, assuming you've set up your git configuration to push changesets without interactive prompts. E.g. to set up pushing to remote servers you could set up SSH public key authentication. Aggregated metadata gets written to a file named :file:`aggregate.meta` at the top-level of a package source and the :ref:`list `, :ref:`search `, and :ref:`info ` all may access this file. Having access to the aggregated metadata in this way is beneficial to all :program:`zkg` users because they then will not have to crawl the set of packages listed in a source in order to obtain this metadata as it will have already been pre-aggregated by the operator of the package source. package-manager-3.0.1/doc/package.rst0000644000175000017500000007655714565163601015551 0ustar rharha.. _Zeek Scripting: https://docs.zeek.org/en/stable/examples/scripting/index.html .. _Zeek Plugins: https://docs.zeek.org/en/stable/devel/plugins.html .. _ZeekControl Plugins: https://github.com/zeek/zeekctl#plugins .. _Semantic Version Specification: https://python-semanticversion.readthedocs.io/en/latest/reference.html#version-specifications-the-spec-class .. _btest: https://github.com/zeek/btest .. _configparser interpolation: https://docs.python.org/3/library/configparser.html#interpolation-of-values How-To: Create a Package ======================== A Zeek package may contain Zeek scripts, Zeek plugins, or ZeekControl plugins. Any number or combination of those components may be included within a single package. The minimum requirement for a package is that it be in its own git repository and contain a metadata file named :file:`zkg.meta` at its top-level that begins with the line:: [package] This is the package's metadata file in INI file format and may contain :ref:`additional fields ` that describe the package as well as how it inter-operates with Zeek, the package manager, or other packages. .. note:: :file:`zkg.meta` is the canonical metadata file name used :program:`since zkg v2.0`. The previous metadata file name of :file:`bro-pkg.meta` is also accepted when no :file:`zkg.meta` exists. .. _package-shorthand-name: Note that the shorthand name for your package that may be used by :ref:`zkg ` and Zeek script :samp:`@load {}` directives will be the last component of its git URL. E.g. a package at ``https://github.com/zeek/foo`` may be referred to as **foo** when using :program:`zkg` and a Zeek script that wants to load all the scripts within that package can use: .. code-block:: zeek @load foo Bootstrapping packages with :program:`zkg` ------------------------------------------ The easiest way to start a new Zeek package is via :program:`zkg` itself: its ``zkg create`` command lets you generate new Zeek packages from the command line. This functionality is available since :program:`since zkg v2.9`. See the :ref:`Walkthroughs ` section for step-by-step processes that show how to manually create packages (e.g. perhaps when using older :program:`zkg` versions). Concepts ~~~~~~~~ :program:`zkg` instantiates new packages from a *package template*. Templates are standalone git repositories. The URL of :program:`zkg`'s default template is https://github.com/zeek/package-template, but you can provide your own. .. note:: At :program:`zkg` configuration time, the ``ZKG_DEFAULT_TEMPLATE`` environment variable lets you override the default, and the ``--template`` argument to ``zkg create`` allows overrides upon instantiation. You can review the template :program:`zkg` will use by default via the ``zkg config`` command's output. A template provides a basic *package* layout, with optional added *features* that enhance the package. For example, the default template lets you add a native-code plugin and support for GitHub actions. Templates are parameterized via :ref:`user variables `. These variables provide the basic configuration required when instantiating the template, for example to give the package a name. A template uses resolved user variables to populate internal *parameters* that the template requires. Think of parameters as derivatives of the user variables, for example to provide different capitalizations or suffixes. A template operates as a :program:`zkg` plugin, including runnable Python code. This code has full control over how a package gets instantiated, defining required user variables and features, and possibly customizing content production. The ``create`` command ~~~~~~~~~~~~~~~~~~~~~~ When using the ``zkg create`` command, you specify an output directory for the new package tree, name the features you'd like to add, and optionally define user variables. :program:`zkg` will prompt you for any variables it still needs to resolve, and guides you through the package creation. A basic invocation might look as follows: .. code-block:: console $ zkg create --packagedir foobar --feature plugin "package-template" requires a "name" value (the name of the package, e.g. "FooBar"): name: Foobar "package-template" requires a "namespace" value (a namespace for the package, e.g. "MyOrg"): namespace: MyOrg The resulting package now resides in the ``foobar`` directory. Unless you provide ``--force``, :program:`zkg` will not overwrite an existing package. When the requested output directory exists, it will prompt for permission to delete the existing directory. After instantiation, the package is immediately installable via :program:`zkg`. You'll see details of how it got generated in its initial commit, and the newly minted ``zkg.meta`` has details of the provided user variables: .. code-block:: console $ cat foobar/zkg.meta ... [template] source = package-template version = master zkg_version = 2.8.0 features = plugin [template_vars] name = Foobar namespace = MyOrg This information is currently informational only, but in the future will enable baselining changes in package templates to assist with package modernization. To keep templates in sync with :program:`zkg` versions, templates employ semantic API versioning. An incompatible template will refuse to load and lead to an according error message. Much like Zeek packages, templates support git-level versioning to accommodate compatibility windows. See the output of ``zkg create --help`` for a complete summary of the available options. Obtaining information about a template ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The best source for the capabilities of a template is its documentation, but to get a quick overview of a given template's features and user variables, consider the ``zkg template info`` command, which summarizes a template in plain text, or in JSON when invoked with the ``--json`` argument. .. _manual-package-creation: Walkthroughs ------------ For historical reference, the following sections cover manual ways of establishing Zeek packages. Pure Zeek Script Package ~~~~~~~~~~~~~~~~~~~~~~~~ #. Create a git repository: .. code-block:: console $ mkdir foo && cd foo && git init #. Create a package metadata file, :file:`zkg.meta`: .. code-block:: console $ echo '[package]' > zkg.meta #. Create a :file:`__load__.zeek` script with example code in it: .. code-block:: console $ echo 'event zeek_init() { print "foo is loaded"; }' > __load__.zeek #. (Optional) Relocate your :file:`__load__.zeek` script to any subdirectory: .. code-block:: console $ mkdir scripts && mv __load__.zeek scripts $ echo 'script_dir = scripts' >> zkg.meta #. Commit everything to git: .. code-block:: console $ git add * && git commit -m 'First commit' #. (Optional) Test that Zeek correctly loads the script after installing the package with :program:`zkg`: .. code-block:: console $ zkg install . $ zeek foo $ zkg remove . #. (Optional) :ref:`Create a release version tag `. See `Zeek Scripting`_ for more information on developing Zeek scripts. Binary Zeek Plugin Package ~~~~~~~~~~~~~~~~~~~~~~~~~~ See `Zeek Plugins`_ for more complete information on developing Zeek plugins, though the following step are the essentials needed to create a package. #. Create a plugin skeleton using :file:`aux*/zeek-aux/plugin-support/init-plugin` from Zeek's source distribution: .. code-block:: console $ init-plugin ./rot13 Demo Rot13 #. Create a git repository .. code-block:: console $ cd rot13 && git init #. Create a package metadata file, :file:`zkg.meta`:: [package] script_dir = scripts/Demo/Rot13 build_command = ./configure && make #. Add example script code: .. code-block:: console $ echo 'event zeek_init() { print "rot13 plugin is loaded"; }' >> scripts/__load__.zeek $ echo 'event zeek_init() { print "rot13 script is loaded"; }' >> scripts/Demo/Rot13/__load__.zeek #. Add an example builtin-function in :file:`src/rot13.bif`: .. code-block:: c++ module Demo; function rot13%(s: string%) : string %{ char* rot13 = copy_string(s->CheckString()); for ( char* p = rot13; *p; p++ ) { char b = islower(*p) ? 'a' : 'A'; *p = (*p - b + 13) % 26 + b; } return make_intrusive(strlen(rot13), rot13); %} #. Commit everything to git: .. code-block:: console $ git add * && git commit -m 'First commit' #. (Optional) Test that Zeek correctly loads the plugin after installing the package with :program:`zkg`: .. code-block:: console $ zkg install . $ zeek rot13 -e 'print Demo::rot13("Hello")' $ zkg remove . #. (Optional) :ref:`Create a release version tag `. ZeekControl Plugin Package ~~~~~~~~~~~~~~~~~~~~~~~~~~ #. Create a git repository: .. code-block:: console $ mkdir foo && cd foo && git init #. Create a package metadata file, :file:`zkg.meta`: .. code-block:: console $ echo '[package]' > zkg.meta #. Create an example ZeekControl plugin, :file:`foo.py`: .. code-block:: python import ZeekControl.plugin from ZeekControl import config class Foo(ZeekControl.plugin.Plugin): def __init__(self): super(Foo, self).__init__(apiversion=1) def name(self): return "foo" def pluginVersion(self): return 1 def init(self): self.message("foo plugin is initialized") return True #. Set the `plugin_dir` metadata field to directory where the plugin is located: .. code-block:: console $ echo 'plugin_dir = .' >> zkg.meta #. Commit everything to git: .. code-block:: console $ git add * && git commit -m 'First commit' #. (Optional) Test that ZeekControl correctly loads the plugin after installing the package with :program:`zkg`: .. code-block:: console $ zkg install . $ zeekctl $ zkg remove . #. (Optional) :ref:`Create a release version tag `. See `ZeekControl Plugins`_ for more information on developing ZeekControl plugins. If you want to distribute a ZeekControl plugin along with a Zeek plugin in the same package, you may need to add the ZeekControl plugin's python script to the ``zeek_plugin_dist_files()`` macro in the :file:`CMakeLists.txt` of the Zeek plugin so that it gets copied into :file:`build/` along with the built Zeek plugin. Or you could also modify your `build_command` to copy it there, but what ultimately matters is that the `plugin_dir` field points to a directory that contains both the Zeek plugin and the ZeekControl plugin. Registering to a Package Source ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Registering a package to a package source is always the following basic steps: #) Create a :ref:`Package Index File ` for your package. #) Add the index file to the package source's git repository. The full process and conventions for submitting to the default package source can be found in the :file:`README` at: https://github.com/zeek/packages .. _metadata-fields: Package Metadata ---------------- See the following sub-sections for a full list of available fields that may be used in :file:`zkg.meta` files. `description` field ~~~~~~~~~~~~~~~~~~~ The description field may be used to give users a general overview of the package and its purpose. The :ref:`zkg list ` will display the first sentence of description fields in the listings it displays. An example :file:`zkg.meta` using a description field:: [package] description = Another example package. The description text may span multiple line: when adding line breaks, just indent the new lines so they are parsed as part of the 'description' value. `aliases` field ~~~~~~~~~~~~~~~ The `aliases` field can be used to specify alternative names for a package. Users can then use :samp:`@load {}` for any alias listed in this field. This may be useful when renaming a package's repository on GitHub while still supporting users that already installed the package under the previous name. For example, if package `foo` were renamed to `foo2`, then the `aliases` for it could be:: [package] aliases = foo2 foo Currently, the order does not matter, but you should specify the canonical/current alias first. The list is delimited by commas or whitespace. If this field is not specified, the default behavior is the same as if using a single alias equal to the package's name. The low-level details of the way this field operates is that, for each alias, it simply creates a symlink of the same name within the directory associated with the ``script_dir`` path in the :ref:`config file `. Available :program:`since bro-pkg v1.5`. `credits` field ~~~~~~~~~~~~~~~ The `credits` field contains a comma-delimited set of author/contributor/maintainer names, descriptions, and/or email addresses. It may be used if you have particular requirements or concerns regarding how authors or contributors for your package are credited in any public listings made by external metadata scraping tools (:program:`zkg` does not itself use this data directly for any functional purpose). It may also be useful as a standardized location for users to get contact/support info in case they encounter problems with the package. For example:: [package] credits = A. Sacker ., JSON support added by W00ter (Acme Corporation) `tags` field ~~~~~~~~~~~~ The `tags` field contains a comma-delimited set of metadata tags that further classify and describe the purpose of the package. This is used to help users better discover and search for packages. The :ref:`zkg search ` command will inspect these tags. An example :file:`zkg.meta` using tags:: [package] tags = zeek plugin, zeekctl plugin, scan detection, intel Suggested Tags ^^^^^^^^^^^^^^ Some ideas for what to put in the `tags` field for packages: - zeek scripting - conn - intel - geolocation - file analysis - sumstats, summary statistics - input - log, logging - notices - ** - ** - signatures - zeek plugin - protocol analyzer - file analyzer - bifs - packet source - packet dumper - input reader - log writer - zeekctl plugin `script_dir` field ~~~~~~~~~~~~~~~~~~ The `script_dir` field is a path relative to the root of the package that contains a file named :file:`__load__.zeek` and possibly other Zeek scripts. The files located in this directory are copied into :file:`{}/packages/{}/`, where `` corresponds to the `script_dir` field of the user's :ref:`config file ` (typically :file:`{}/share/zeek/site`). When the package is :ref:`loaded `, an :samp:`@load {}` directive is added to :file:`{}/packages/packages.zeek`. You may place any valid Zeek script code within :file:`__load__.zeek`, but a package that contains many Zeek scripts will typically have :file:`__load__.zeek` just contain a list of ``@load`` directives to load other Zeek scripts within the package. E.g. if you have a package named **foo** installed, then it's :file:`__load__.zeek` will be what Zeek loads when doing ``@load foo`` or running ``zeek foo`` on the command-line. An example :file:`zkg.meta`:: [package] script_dir = scripts For a :file:`zkg.meta` that looks like the above, the package should have a file called :file:`scripts/__load__.zeek`. If the `script_dir` field is not present in :file:`zkg.meta`, it defaults to checking the top-level directory of the package for a :file:`__load__.zeek` script. If it's found there, :program:`zkg` use the top-level package directory as the value for `script_dir`. If it's not found, then :program:`zkg` assumes the package contains no Zeek scripts (which may be the case for some plugins). `plugin_dir` field ~~~~~~~~~~~~~~~~~~ The `plugin_dir` field is a path relative to the root of the package that contains either pre-built `Zeek Plugins`_, `ZeekControl Plugins`_, or both. An example :file:`zkg.meta`:: [package] script_dir = scripts plugin_dir = plugins For the above example, Zeek and ZeekControl will load any plugins found in the installed package's :file:`plugins/` directory. If the `plugin_dir` field is not present in :file:`zkg.meta`, it defaults to a directory named :file:`build/` at the top-level of the package. This is the default location where Zeek binary plugins get placed when building them from source code (see the `build_command field`_). This field may also be set to the location of a tarfile that has a single top- level directory inside it containing the Zeek plugin. The default CMake skeleton for Zeek plugins produces such a tarfile located at :file:`build/_.tgz`. This is a good choice to use for packages that will be published to a wider audience as installing from this tarfile contains the minimal set of files needed for the plugin to work whereas some extra files will get installed to user systems if the `plugin_dir` uses the default :file:`build/` directory. `executables` field ~~~~~~~~~~~~~~~~~~~ The `executables` field is a whitespace-delimited list of shell scripts or other executables that the package provides. The package manager will make these executables available inside the user's :file:`bin_dir` directory as specified in the :ref:`config file `. An example :file:`zkg.meta`, if the ``Rot13`` example plugin were also building an executable ``a.out``:: [package] script_dir = scripts/Demo/Rot13 build_command = ./configure && make executables = build/a.out The package manager makes executables available by maintaining symbolic links referring from :file:`bin_dir` to the actual files. Available :program:`since bro-pkg v2.8`. `build_command` field ~~~~~~~~~~~~~~~~~~~~~ The `build_command` field is an arbitrary shell command that the package manager will run before installing the package. This is useful for distributing `Zeek Plugins`_ as source code and having the package manager take care of building it on the user's machine before installing the package. An example :file:`zkg.meta`:: [package] script_dir = scripts/Demo/Rot13 build_command = ./configure && make The default CMake skeleton for Zeek plugins will use :file:`build/` as the directory for the final/built version of the plugin, which matches the defaulted value of the omitted `plugin_dir` metadata field. The `script_dir` field is set to the location where the author has placed custom scripts for their plugin. When a package has both a Zeek plugin and Zeek script components, the "plugin" part is always unconditionally loaded by Zeek, but the "script" components must either be explicitly loaded (e.g. :samp:`@load {}`) or the package marked as :ref:`loaded `. .. _metadata-interpolation: Value Interpolation ^^^^^^^^^^^^^^^^^^^ The `build_command field`_ may reference the settings any given user has in their customized :ref:`package manager config file `. For example, if a metadata field's value contains the ``%(zeek_dist)s`` string, then :program:`zkg` operations that use that field will automatically substitute the actual value of `zeek_dist` that the user has in their local config file. Note the trailing 's' character at the end of the interpolation string, ``%(zeek_dist)s``, is intended/necessary for all such interpolation usages. Besides the `zeek_dist` config key, any key inside the `user_vars` sections of their :ref:`package manager config file ` that matches the key of an entry in the package's `user_vars field`_ will be interpolated. Another pre-defined config key is `package_base`, which points to the top-level directory where :program:`zkg` stores all installed packages (i.e. clones of each package's git repository). This can be used to gain access to the content of another package that was installed as a dependency. Note that `package_base` is only available :program:`since zkg v2.3` Internally, the value substitution and metadata parsing is handled by Python's `configparser interpolation`_. See its documentation if you're interested in the details of how the interpolation works. .. _user-vars: `user_vars` field ~~~~~~~~~~~~~~~~~ The `user_vars` field is used to solicit feedback from users for use during execution of the `build_command field`_. An example :file:`zkg.meta`:: [package] build_command = ./configure --with-librdkafka=%(LIBRDKAFKA_ROOT)s --with-libdub=%(LIBDBUS_ROOT)s && make user_vars = LIBRDKAFKA_ROOT [/usr] "Path to librdkafka installation" LIBDBUS_ROOT [/usr] "Path to libdbus installation" The format of the field is a sequence entries of the format:: key [value] "description" The `key` is the string that should match what you want to be interpolated within the `build_command field`_. The `value` is provided as a convenient default value that you'd typically expect to work for most users. The `description` is provided as an explanation for what the value will be used for. Here's what a typical user would see:: $ zkg install zeek-test-package The following packages will be INSTALLED: zeek/jsiwek/zeek-test-package (1.0.5) Proceed? [Y/n] y zeek/jsiwek/zeek-test-package asks for LIBRDKAFKA_ROOT (Path to librdkafka installation) ? [/usr] /usr/local Saved answers to config file: /Users/jon/.zkg/config Installed "zeek/jsiwek/zeek-test-package" (master) Loaded "zeek/jsiwek/zeek-test-package" The :program:`zkg` command will iterate over the `user_vars` field of all packages involved in the operation and prompt the user to provide a value that will work for their system. If a user is using the ``--force`` option to :program:`zkg` commands or they are using the Python API directly, it will first look within the `user_vars` section of the user's :ref:`package manager config file ` and, if it can't find the key there, it will fallback to use the default value from the package's metadata. In any case, the user may choose to supply the value of a `user_vars` key via an environment variable, in which case, prompts are skipped for any keys located in the environment. The user may also provide `user_vars` via ``--user-var NAME=VAL`` command-line arguments. These arguments are given priority over environment variables, which in turn take precedence over any values in the user's :ref:`package manager config file `. Available :program:`since bro-pkg v1.1`. `test_command` field ~~~~~~~~~~~~~~~~~~~~ The `test_command` field is an arbitrary shell command that the package manager will run when a user either manually runs the :ref:`test command ` or before the package is installed or upgraded. An example :file:`zkg.meta`:: [package] test_command = cd testing && btest -d tests The recommended test framework for writing package unit tests is `btest`_. See its documentation for further explanation and examples. .. note:: :program:`zkg` version 2.12.0 introduced two improvements to `test_command`: - :program:`zkg` now honors package dependencies at test time, meaning that if your package depends on another during testing, :program:`zkg` will ensure that the dependency is built and available to your package tests. Only when all testing succeeds does the full set of new packages get installed. - The `test_command` now supports value interpolation similarly to the `build_command field`_. `config_files` field ~~~~~~~~~~~~~~~~~~~~ The `config_files` field may be used to specify a list of files that users are intended to directly modify after installation. Then, on operations that would otherwise destroy a user's local modifications to a config file, such as upgrading to a newer package version, :program:`zkg` can instead save a backup and possibly prompt the user to review the differences. An example :file:`zkg.meta`:: [package] script_dir = scripts config_files = scripts/foo_config.zeek, scripts/bar_config.zeek The value of `config_files` is a comma-delimited string of config file paths that are relative to the root directory of the package. Config files should either be located within the `script_dir` or `plugin_dir`. .. _package-dependencies: `depends` field ~~~~~~~~~~~~~~~ The `depends` field may be used to specify a list of dependencies that the package requires. An example :file:`zkg.meta`:: [package] depends = zeek >=2.5.0 foo * https://github.com/zeek/bar >=2.0.0 package_source/path/bar branch=name_of_git_branch The field is a list of dependency names and their version requirement specifications. A dependency name may be either `zeek`, `zkg`, a full git URL of the package, or a :ref:`package shorthand name `. - The special `zeek` dependency refers not to a package, but the version of Zeek that the package requires in order to function. If the user has :program:`zeek-config` in their :envvar:`PATH` when installing/upgrading a package that specifies a `zeek` dependency, then :program:`zkg` will enforce that the requirement is satisfied. - The special `zkg` dependency refers to the version of the package manager that is required by the package. E.g. if a package takes advantage of new features that are not present in older versions of the package manager, then it should indicate that so users of those old version will see an error message an know to upgrade instead of seeing a cryptic error/exception, or worse, seeing no errors, but without the desired functionality being performed. - The full git URL may be directly specified in the `depends` metadata if you want to force the dependency to always resolve to a single, canonical git repository. Typically this is the safe approach to take when listing package dependencies and for publicly visible packages. - When using shorthand package dependency names, the user's :program:`zkg` will try to resolve the name into a full git URL based on the package sources they have configured. Typically this approach may be most useful for internal or testing environments. A version requirement may be either a git branch name or a semantic version specification. When using a branch as a version requirement, prefix the branchname with ``branch=``, else see the `Semantic Version Specification`_ documentation for the complete rule set of acceptable version requirement strings. Here's a summary: - ``*``: any version (this will also satisfy/match on git branches) - ``<1.0.0``: versions less than 1.0.0 - ``<=1.0.0``: versions less than or equal to 1.0.0 - ``>1.0.0``: versions greater than 1.0.0 - ``>=1.0.0``: versions greater than or equal to 1.0.0 - ``==1.0.0``: exactly version 1.0.0 - ``!=1.0.0``: versions not equal to 1.0.0 - ``^1.3.4``: versions between 1.3.4 and 2.0.0 (not including 2.0.0) - ``~1.2.3``: versions between 1.2.3 and 1.3.0 (not including 1.3.0) - ``~=2.2``: versions between 2.2.0 and 3.0.0 (not included 3.0.0) - ``~=1.4.5``: versions between 1.4.5 and 1.5.0 (not including 3.0.0) - Any of the above may be combined by a separating comma to logically "and" the requirements together. E.g. ``>=1.0.0,<2.0.0`` means "greater or equal to 1.0.0 and less than 2.0.0". Note that these specifications are strict semantic versions. Even if a given package chooses to use the ``vX.Y.Z`` format for its :ref:`git version tags `, do not use the 'v' prefix in the version specifications here as that is not part of the semantic version. `external_depends` field ~~~~~~~~~~~~~~~~~~~~~~~~ The `external_depends` field follows the same format as the :ref:`depends field `, but the dependency names refer to external/third-party software packages. E.g. these would be set to typical package names you'd expect the package manager from any given operating system to use, like 'libpng-dev'. The version specification should also generally be given in terms of semantic versioning where possible. In any case, the name and version specification for an external dependency are only used for display purposes -- to help users understand extra pre-requisites that are needed for proceeding with package installation/upgrades. Available :program:`since bro-pkg v1.1`. `suggests` field ~~~~~~~~~~~~~~~~ The `suggests` field follows the same format as the :ref:`depends field `, but it's used for specifying optional packages that users may want to additionally install. This is helpful for suggesting complementary packages that aren't strictly required for the suggesting package to function properly. A package in `suggests` is functionaly equivalent to a package in `depends` except in the way it's presented to users in various prompts during :program:`zkg` operations. Users also have the option to ignore suggestions by supplying an additional ``--nosuggestions`` flag to :program:`zkg` commands. Available :program:`since bro-pkg v1.3`. .. _package-versioning: Package Versioning ------------------ Creating New Package Release Versions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Package's should use git tags for versioning their releases. Use the `Semantic Versioning `_ numbering scheme here. For example, to create a new tag for a package: .. code-block:: console $ git tag -a 1.0.0 -m 'Release 1.0.0' The tag name may also be of the ``vX.Y.Z`` form (prefixed by 'v'). Choose whichever you prefer. Then, assuming you've already set up a public/remote git repository (e.g. on GitHub) for your package, remember to push the tag to the remote repository: .. code-block:: console $ git push --tags Alternatively, if you expect to have a simple development process for your package, you may choose to not create any version tags and just always make commits directly to your package's default branch (typically named *main* or *master*). Users will receive package updates differently depending on whether you decide to use release version tags or not. See the :ref:`package upgrade process ` documentation for more details on the differences. .. _package-upgrade-process: Package Upgrade Process ~~~~~~~~~~~~~~~~~~~~~~~ The :ref:`install command ` will either install a stable release version or the latest commit on a specific git branch of a package. The default installation behavior of :program:`zkg` is to look for the latest release version tag and install that. If there are no such version tags, it will fall back to installing the latest commit of the package's default branch (typically named *main* or *master*) Upon installing a package via a :ref:`git version tag `, the :ref:`upgrade command ` will only upgrade the local installation of that package if a greater version tag is available. In other words, you only receive stable release upgrades for packages installed in this way. Upon installing a package via a git branch name, the :ref:`upgrade command ` will upgrade the local installation of the package whenever a new commit becomes available at the end of the branch. This method of tracking packages is suitable for testing out development/experimental versions of packages. If a package was installed via a specific commit hash, then the package will never be eligible for automatic upgrades. package-manager-3.0.1/doc/api/0000755000175000017500000000000014565163601014151 5ustar rharhapackage-manager-3.0.1/doc/api/source.rst0000644000175000017500000000020514565163601016200 0ustar rharhazeekpkg.source module ===================== .. automodule:: zeekpkg.source :members: :undoc-members: :show-inheritance: package-manager-3.0.1/doc/api/package.rst0000644000175000017500000000021014565163601016267 0ustar rharhazeekpkg.package module ====================== .. automodule:: zeekpkg.package :members: :undoc-members: :show-inheritance: package-manager-3.0.1/doc/api/uservar.rst0000644000175000017500000000021014565163601016363 0ustar rharhazeekpkg.uservar module ====================== .. automodule:: zeekpkg.uservar :members: :undoc-members: :show-inheritance: package-manager-3.0.1/doc/api/manager.rst0000644000175000017500000000021014565163601016306 0ustar rharhazeekpkg.manager module ====================== .. automodule:: zeekpkg.manager :members: :undoc-members: :show-inheritance: package-manager-3.0.1/doc/api/index.rst0000644000175000017500000000036514565163601016016 0ustar rharhaPython API Reference ==================== .. automodule:: zeekpkg The following Python modules are all provided as part of the ``zeekpkg`` public interface: .. toctree:: :maxdepth: 2 manager package source template uservar package-manager-3.0.1/doc/api/template.rst0000644000175000017500000000021314565163601016512 0ustar rharhazeekpkg.template module ======================= .. automodule:: zeekpkg.template :members: :undoc-members: :show-inheritance: package-manager-3.0.1/doc/man/0000755000175000017500000000000014565163601014153 5ustar rharhapackage-manager-3.0.1/doc/man/zkg.10000644000175000017500000007402614565163601015041 0ustar rharha.\" Man page generated from reStructuredText. . . .nr rst2man-indent-level 0 . .de1 rstReportMargin \\$1 \\n[an-margin] level \\n[rst2man-indent-level] level margin: \\n[rst2man-indent\\n[rst2man-indent-level]] - \\n[rst2man-indent0] \\n[rst2man-indent1] \\n[rst2man-indent2] .. .de1 INDENT .\" .rstReportMargin pre: . RS \\$1 . nr rst2man-indent\\n[rst2man-indent-level] \\n[an-margin] . nr rst2man-indent-level +1 .\" .rstReportMargin post: .. .de UNINDENT . RE .\" indent \\n[an-margin] .\" old: \\n[rst2man-indent\\n[rst2man-indent-level]] .nr rst2man-indent-level -1 .\" new: \\n[rst2man-indent\\n[rst2man-indent-level]] .in \\n[rst2man-indent\\n[rst2man-indent-level]]u .. .TH "ZKG" "1" "Feb 20, 2024" "3.0.1" "Zeek Package Manager" .SH NAME zkg \- Zeek Package Manager .sp A command\-line package manager for Zeek. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg [\-h] [\-\-version] [\-\-configfile FILE | \-\-user] [\-\-verbose] [\-\-extra\-source NAME=URL] {test,install,bundle,unbundle,remove,uninstall,purge,refresh,upgrade,load,unload,pin,unpin,list,search,info,config,autoconfig,env,create,template} ... .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-version show program\(aqs version number and exit .TP .B \-\-configfile Path to Zeek Package Manager config file. Precludes \-\-user. .sp See \fI\%Config File\fP\&. .TP .B \-\-user=False Store all state in user\(aqs home directory. Precludes \-\-configfile. .TP .B \-\-verbose=0\fP,\fB \-v=0 Increase program output for debugging. Use multiple times for more output (e.g. \-vv). .TP .B \-\-extra\-source Add an extra source. .UNINDENT .UNINDENT .sp Environment Variables: .INDENT 0.0 .INDENT 3.5 \fBZKG_CONFIG_FILE\fP: Same as \fB\-\-configfile\fP option, but has less precedence. \fBZKG_DEFAULT_SOURCE\fP: The default package source to use (normally \fI\%https://github.com/zeek/packages\fP). \fBZKG_DEFAULT_TEMPLATE\fP: The default package template to use (normally \fI\%https://github.com/zeek/package\-template\fP). .UNINDENT .UNINDENT .SH COMMANDS .SS test .sp Runs the unit tests for the specified Zeek packages. In most cases, the "zeek" and "zeek\-config" programs will need to be in PATH before running this command. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg test [\-h] [\-\-version VERSION] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-version The version of the package to test. Only one package may be specified at a time when using this flag. A version tag, branch name, or commit hash may be specified here. If the package name refers to a local git repo with a working tree, then its currently active branch is used. The default for other cases is to use the latest version tag, or if a package has none, the default branch, like "main" or "master". .UNINDENT .UNINDENT .SS install .sp Installs packages from a configured package source or directly from a git URL. After installing, the package is marked as being "loaded" (see the \fBload\fP command). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg install [\-h] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-version VERSION] [\-\-force] [\-\-user\-var NAME=VAL] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-skiptests=False Skip running unit tests for packages before installation. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .TP .B \-\-nosuggestions=False Skip automatically installing suggested packages. .TP .B \-\-version The version of the package to install. Only one package may be specified at a time when using this flag. A version tag, branch name, or commit hash may be specified here. If the package name refers to a local git repo with a working tree, then its currently active branch is used. The default for other cases is to use the latest version tag, or if a package has none, the default branch, like "main" or "master". .TP .B \-\-force=False Don\(aqt prompt for confirmation or user variables. .TP .B \-\-user\-var A user variable assignment. This avoids prompting for input and lets you provide a value when using \-\-force. Use repeatedly as needed for multiple values. .UNINDENT .UNINDENT .SS remove .sp Unloads (see the \fBunload\fP command) and uninstalls a previously installed package. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg remove [\-h] [\-\-force] [\-\-nodeps] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .UNINDENT .UNINDENT .sp \fBNOTE:\fP .INDENT 0.0 .INDENT 3.5 You may also say \fBuninstall\fP\&. .UNINDENT .UNINDENT .SS purge .sp Unloads (see the \fBunload\fP command) and uninstalls all previously installed packages. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg purge [\-h] [\-\-force] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .UNINDENT .UNINDENT .SS bundle .sp This command creates a bundle file containing a collection of Zeek packages. If \fB\-\-manifest\fP is used, the user supplies the list of packages to put in the bundle, else all currently installed packages are put in the bundle. A bundle file can be unpacked on any target system, resulting in a repeatable/specific set of packages being installed on that target system (see the \fBunbundle\fP command). This command may be useful for those that want to manage packages on a system that otherwise has limited network connectivity. E.g. one can use a system with an internet connection to create a bundle, transport that bundle to the target machine using whatever means are appropriate, and finally unbundle/install it on the target machine. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg bundle [\-h] [\-\-force] [\-\-nodeps] [\-\-nosuggestions] [\-\-manifest MANIFEST [MANIFEST ...] \-\-] filename.bundle .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B filename.bundle The path of the bundle file to create. It will be overwritten if it already exists. Note that if \-\-manifest is used before this filename is specified, you should use a double\-dash, \-\-, to first terminate that argument list. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks creating a bundle of packages that is in a broken or unusable state. .TP .B \-\-nosuggestions=False Skip automatically bundling suggested packages. .TP .B \-\-manifest This may either be a file name or a list of packages to include in the bundle. If a file name is supplied, it should be in INI format with a single \(ga\(ga[bundle]\(ga\(ga section. The keys in that section correspond to package names and their values correspond to git version tags, branch names, or commit hashes. The values may be left blank to indicate that the latest available version should be used. .UNINDENT .UNINDENT .SS unbundle .sp This command unpacks a bundle file formerly created by the \fBbundle\fP command and installs all the packages contained within. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg unbundle [\-h] [\-\-replace] [\-\-force] [\-\-user\-var NAME=VAL] filename.bundle .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B filename.bundle The path of the bundle file to install. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-replace=False Using this flag first removes all installed packages before then installing the packages from the bundle. .TP .B \-\-force=False Don\(aqt prompt for confirmation or user variables. .TP .B \-\-user\-var A user variable assignment. This avoids prompting for input and lets you provide a value when using \-\-force. Use repeatedly as needed for multiple values. .UNINDENT .UNINDENT .SS refresh .sp Retrieve latest package metadata from sources and checks whether any installed packages have available upgrades. Note that this does not actually upgrade any packages (see the \fBupgrade\fP command for that). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg refresh [\-h] [\-\-aggregate] [\-\-fail\-on\-aggregate\-problems] [\-\-push] [\-\-sources SOURCES [SOURCES ...]] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-aggregate=False Crawls the urls listed in package source zkg.index files and aggregates the metadata found in their zkg.meta (or legacy bro\-pkg.meta) files. The aggregated metadata is stored in the local clone of the package source that zkg uses internally for locating package metadata. For each package, the metadata is taken from the highest available git version tag or the default branch, like "main" or "master", if no version tags exist .TP .B \-\-fail\-on\-aggregate\-problems=False When using \-\-aggregate, exit with error when any packages trigger metadata problems. Normally such problems only cause a warning. .TP .B \-\-push=False Push all local changes to package sources to upstream repos .TP .B \-\-sources A list of package source names to operate on. If this argument is not used, then the command will operate on all configured sources. .UNINDENT .UNINDENT .SS upgrade .sp Uprades the specified package(s) to latest available version. If no specific packages are specified, then all installed packages that are outdated and not pinned are upgraded. For packages that are installed with \fB\-\-version\fP using a git branch name, the package is updated to the latest commit on that branch, else the package is updated to the highest available git version tag. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg upgrade [\-h] [\-\-skiptests] [\-\-nodeps] [\-\-nosuggestions] [\-\-force] [\-\-user\-var NAME=VAL] [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-skiptests=False Skip running unit tests for packages before installation. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .TP .B \-\-nosuggestions=False Skip automatically installing suggested packages. .TP .B \-\-force=False Don\(aqt prompt for confirmation or user variables. .TP .B \-\-user\-var A user variable assignment. This avoids prompting for input and lets you provide a value when using \-\-force. Use repeatedly as needed for multiple values. .UNINDENT .UNINDENT .SS load .sp The Zeek Package Manager keeps track of all packages that are marked as "loaded" and maintains a single Zeek script that, when loaded by Zeek (e.g. via \fB@load packages\fP), will load the scripts from all "loaded" packages at once. This command adds a set of packages to the "loaded packages" list. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg load [\-h] [\-\-nodeps] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package Name(s) of package(s) to load. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .UNINDENT .UNINDENT .SS unload .sp The Zeek Package Manager keeps track of all packages that are marked as "loaded" and maintains a single Zeek script that, when loaded by Zeek, will load the scripts from all "loaded" packages at once. This command removes a set of packages from the "loaded packages" list. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg unload [\-h] [\-\-force] [\-\-nodeps] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip the confirmation prompt. .TP .B \-\-nodeps=False Skip all dependency resolution/checks. Note that using this option risks putting your installed package collection into a broken or unusable state. .UNINDENT .UNINDENT .SS pin .sp Pinned packages are ignored by the \fBupgrade\fP command. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg pin [\-h] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .UNINDENT .SS unpin .sp Packages that are not pinned are automatically upgraded by the \fBupgrade\fP command .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg unpin [\-h] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". .UNINDENT .UNINDENT .SS list .sp Outputs a list of packages that match a given category. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg list [\-h] [\-\-nodesc] [\-\-include\-builtin] [{all,installed,not_installed,loaded,unloaded,outdated}] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B category Package category used to filter listing. .sp Possible choices: all, installed, not_installed, loaded, unloaded, outdated .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-nodesc=False Do not display description text, just the package name(s). .TP .B \-\-include\-builtin=False Also output packages that Zeek has built\-in. By default these are not shown. .UNINDENT .UNINDENT .SS search .sp Perform a substring search on package names and metadata tags. Surround search text with slashes to indicate it is a regular expression (e.g. \fB/text/\fP). .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg search [\-h] search_text [search_text ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B search_text The text(s) or pattern(s) to look for. .UNINDENT .UNINDENT .SS info .sp Shows detailed information/metadata for given packages. If the package is currently installed, additional information about the status of it is displayed. E.g. the installed version or whether it is currently marked as "pinned" or "loaded." .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg info [\-h] [\-\-version VERSION] [\-\-nolocal] [\-\-include\-builtin] [\-\-json] [\-\-jsonpretty SPACES] [\-\-allvers] package [package ...] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B package The name(s) of package(s) to operate on. The package may be named in several ways. If the package is part of a package source, it may be referred to by the base name of the package (last component of git URL) or its path within the package source. If two packages in different package sources have conflicting paths, then the package source name may be prepended to the package path to resolve the ambiguity. A full git URL may also be used to refer to a package that does not belong to a source. E.g. for a package source called "zeek" that has a package named "foo" located in "alice/zkg.index", the following names work: "foo", "alice/foo", "zeek/alice/foo". If a single name is given and matches one of the same categories as the "list" command, then it is automatically expanded to be the names of all packages which match the given category. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-version The version of the package metadata to inspect. A version tag, branch name, or commit hash and only one package at a time may be given when using this flag. If unspecified, the behavior depends on whether the package is currently installed. If installed, the metadata will be pulled from the installed version. If not installed, the latest version tag is used, or if a package has no version tags, the default branch, like "main" or "master", is used. .TP .B \-\-nolocal=False Do not read information from locally installed packages. Instead read info from remote GitHub. .TP .B \-\-include\-builtin=False Also output packages that Zeek has built\-in. By default these are not shown. .TP .B \-\-json=False Output package information as JSON. .TP .B \-\-jsonpretty Optional number of spaces to indent for pretty\-printed JSON output. .TP .B \-\-allvers=False When outputting package information as JSON, show metadata for all versions. This option can be slow since remote repositories may be cloned multiple times. Also, installed packages will show metadata only for the installed version unless the \-\-nolocal option is given. .UNINDENT .UNINDENT .SS config .sp The default output of this command is a valid package manager config file that corresponds to the one currently being used, but also with any defaulted field values filled in. This command also allows for only the value of a specific field to be output if the name of that field is given as an argument to the command. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg config [\-h] [{all,sources,user_vars,state_dir,script_dir,plugin_dir,bin_dir,zeek_dist}] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B config_param Name of a specific config file field to output. .sp Possible choices: all, sources, user_vars, state_dir, script_dir, plugin_dir, bin_dir, zeek_dist .UNINDENT .UNINDENT .SS autoconfig .sp The output of this command is a valid package manager config file that is generated by using the \fBzeek\-config\fP script that is installed along with Zeek. It is the suggested configuration to use for most Zeek installations. For this command to work, the \fBzeek\-config\fP script must be in \fBPATH\fP, unless the \-\-user option is given, in which case this creates a config that does not touch the Zeek installation. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg autoconfig [\-h] [\-\-force] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-force=False Skip any confirmation prompt. .UNINDENT .UNINDENT .SS env .sp This command returns shell commands that, when executed, will correctly set \fBZEEKPATH\fP and \fBZEEK_PLUGIN_PATH\fP to use scripts and plugins from packages installed by the package manager. For this command to function properly, either have the \fBzeek\-config\fP script (installed by zeek) in \fBPATH\fP, or have the \fBZEEKPATH\fP and \fBZEEK_PLUGIN_PATH\fP environment variables already set so this command can append package\-specific paths to them. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg env [\-h] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .UNINDENT .SS create .sp This command creates a new Zeek package in the directory provided via \-\-packagedir. If this directory exists, zkg will not modify it unless you provide \-\-force. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg create [\-h] \-\-packagedir DIR [\-\-version VERSION] [\-\-features FEATURE [FEATURE ...]] [\-\-template URL] [\-\-force] [\-\-user\-var NAME=VAL] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Options: .INDENT 7.0 .TP .B \-\-packagedir Output directory into which to produce the new package. Required. .TP .B \-\-version The template version to use. A version tag, branch name, or commit hash may be specified here. If \-\-template refers to a local git repo with a working tree, then zkg uses it as\-is and the version is ignored. The default for other cases is to use the latest version tag, or if a template has none, the default branch, like "main" or "master". .TP .B \-\-features Additional features to include in your package. Use the \(ga\(gatemplate info\(ga\(ga command for information about available features. .TP .B \-\-template By default, zkg uses its own package template. This makes it select an alternative. .TP .B \-\-force=False Don\(aqt prompt for confirmation or user variables. .TP .B \-\-user\-var A user variable assignment. This avoids prompting for input and lets you provide a value when using \-\-force. Use repeatedly as needed for multiple values. .UNINDENT .UNINDENT .SS template info .sp This command shows versions and supported features for a given package. .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C usage: zkg template info [\-h] [\-\-json] [\-\-jsonpretty SPACES] [\-\-version VERSION] [URL] .ft P .fi .UNINDENT .UNINDENT .INDENT 0.0 .TP .B Positional arguments: .INDENT 7.0 .TP .B URL URL of a package template repository, or local path to one. When not provided, the configured default template is used. .UNINDENT .TP .B Options: .INDENT 7.0 .TP .B \-\-json=False Output template information as JSON. .TP .B \-\-jsonpretty Optional number of spaces to indent for pretty\-printed JSON output. .TP .B \-\-version The template version to report on. A version tag, branch name, or commit hash may be specified here. If the selected template refers to a local git repo, the version is ignored. The default for other cases is to use the latest version tag, or if a template has none, the default branch, like "main" or "master". .UNINDENT .UNINDENT .SH CONFIG FILE .sp The \fBzkg\fP command\-line tool uses an INI\-format config file to allow users to customize their \fI\%Package Sources\fP, \fI\%Package\fP installation paths, Zeek executable/source paths, and other \fBzkg\fP options. .sp See the default/example config file below for explanations of the available options and how to customize them: .INDENT 0.0 .INDENT 3.5 .sp .nf .ft C # This is an example config file for zkg to explain what # settings are possible as well as their default values. # The order of precedence for how zkg finds/reads config files: # # (1) zkg \-\-configfile=/path/to/custom/config # (2) the ZKG_CONFIG_FILE environment variable # (3) a config file located at $HOME/.zkg/config # (4) if none of the above exist, then zkg uses builtin/default # values for all settings shown below [sources] # The default package source repository from which zkg fetches # packages. The default source may be removed, changed, or # additional sources may be added as long as they use a unique key # and a value that is a valid git URL. The git URL may also use a # suffix like "@branch\-name" where "branch\-name" is the name of a real # branch to checkout (as opposed to the default branch, which is typically # "main" or "master"). You can override the package source zkg puts # in new config files (e.g. "zkg autoconfig") by setting the # ZKG_DEFAULT_SOURCE environment variable. zeek = https://github.com/zeek/packages [paths] # Directory where source repositories are cloned, packages are # installed, and other package manager state information is # maintained. If left blank or with \-\-user this defaults to # $HOME/.zkg. In Zeek\-bundled installations, it defaults to # /var/lib/zkg/. state_dir = # The directory where package scripts are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "script_dir" option of each package\(aqs zkg.meta # (or legacy bro\-pkg.meta) file there. # If left blank or with \-\-user this defaults to /script_dir. # In Zeek\-bundled installations, it defaults to # /share/zeek/site. # If you decide to change this location after having already # installed packages, zkg will automatically relocate them # the next time you run any zkg command. script_dir = # The directory where package plugins are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "plugin_dir" option of each package\(aqs zkg.meta # (or legacy bro\-pkg.meta) file there. # If left blank or with \-\-user this defaults to /plugin_dir. # In Zeek\-bundled installations, it defaults to # /lib/zeek/plugins. # If you decide to change this location after having already # installed packages, zkg will automatically relocate them # the next time you run any zkg command. plugin_dir = # The directory where executables from packages are linked into upon # installation. If left blank or with \-\-user this defaults to /bin. # In Zeek\-bundled installations, it defaults to /bin. # If you decide to change this location after having already # installed packages, zkg will automatically relocate them # the next time you run any zkg command. bin_dir = # The directory containing Zeek distribution source code. This is only # needed when installing packages that contain Zeek plugins that are # not pre\-built. This value is generally not needed by most users other # than plugin developers anymore. zeek_dist = [templates] # The URL of the package template repository that the "zkg create" command # will instantiate by default. default = https://github.com/zeek/package\-template [user_vars] # For any key in this section that is matched for value interpolation # in a package\(aqs zkg.meta (or legacy bro\-pkg.meta) file, the corresponding # value is substituted during execution of the package\(aqs \(gabuild_command\(ga. # This section is typically automatically populated with the # the answers supplied during package installation prompts # and, as a convenience feature, used to recall the last\-used settings # during subsequent operations (e.g. upgrades) on the same package. .ft P .fi .UNINDENT .UNINDENT .SH AUTHOR The Zeek Project .SH COPYRIGHT 2019, The Zeek Project .\" Generated by docutils manpage writer. . package-manager-3.0.1/doc/overview.rst0000644000175000017500000000217614565163601016006 0ustar rharha.. _Zeek: https://www.zeek.org .. _Zeek package source: https://github.com/zeek/packages .. _documentation: https://docs.zeek.org/projects/package-manager Zeek Package Manager ==================== The Zeek Package Manager makes it easy for Zeek users to install and manage third party scripts as well as plugins for Zeek and ZeekControl. The command-line tool is preconfigured to download packages from the `Zeek package source`_ , a GitHub repository that has been set up such that any developer can request their Zeek package be included. See the ``README`` file of that repository for information regarding the package submission process. .. note:: It's left up to users to decide for themselves via code review, GitHub comments/stars, or other metrics whether any given package is trustworthy as there is no implied guarantees that it's secure just because it's been accepted into the default package source. See the package manager documentation_ for further usage information, how-to guides, and walkthroughs. For offline reading, it's also available in the ``doc/`` directory of the source code distribution. package-manager-3.0.1/doc/index.rst0000644000175000017500000000022014565163601015233 0ustar rharha.. include:: overview.rst .. toctree:: :hidden: :numbered: quickstart zkg package source api/index developers package-manager-3.0.1/doc/ext/0000755000175000017500000000000014565163601014200 5ustar rharhapackage-manager-3.0.1/doc/ext/sphinxarg/0000755000175000017500000000000014565163601016203 5ustar rharhapackage-manager-3.0.1/doc/ext/sphinxarg/parser.py0000644000175000017500000001105714565163601020055 0ustar rharhafrom argparse import _HelpAction, _SubParsersAction import re class NavigationException(Exception): pass def parser_navigate(parser_result, path, current_path=None): if isinstance(path, str): if path == '': return parser_result path = re.split(r'\s+', path) current_path = current_path or [] if len(path) == 0: return parser_result if 'children' not in parser_result: raise NavigationException( 'Current parser have no children elements. (path: %s)' % ' '.join(current_path)) next_hop = path.pop(0) for child in parser_result['children']: # identifer is only used for aliased subcommands identifier = child['identifier'] if 'identifier' in child else child['name'] if identifier == next_hop: current_path.append(next_hop) return parser_navigate(child, path, current_path) raise NavigationException( 'Current parser have no children element with name: %s (path: %s)' % ( next_hop, ' '.join(current_path))) def _try_add_parser_attribute(data, parser, attribname): attribval = getattr(parser, attribname, None) if attribval is None: return if not isinstance(attribval, str): return if len(attribval) > 0: data[attribname] = attribval def _format_usage_without_prefix(parser): """ Use private argparse APIs to get the usage string without the 'usage: ' prefix. """ fmt = parser._get_formatter() fmt.add_usage(parser.usage, parser._actions, parser._mutually_exclusive_groups, prefix='') return fmt.format_help().strip() def parse_parser(parser, data=None, **kwargs): if data is None: data = { 'name': '', 'usage': parser.format_usage().strip(), 'bare_usage': _format_usage_without_prefix(parser), 'prog': parser.prog, } _try_add_parser_attribute(data, parser, 'description') _try_add_parser_attribute(data, parser, 'epilog') for action in parser._get_positional_actions(): if isinstance(action, _HelpAction): continue if isinstance(action, _SubParsersAction): helps = {} for item in action._choices_actions: helps[item.dest] = item.help # commands which share an existing parser are an alias, # don't duplicate docs subsection_alias = {} subsection_alias_names = set() for name, subaction in action._name_parser_map.items(): if subaction not in subsection_alias: subsection_alias[subaction] = [] else: subsection_alias[subaction].append(name) subsection_alias_names.add(name) for name, subaction in action._name_parser_map.items(): if name in subsection_alias_names: continue subalias = subsection_alias[subaction] subaction.prog = '%s %s' % (parser.prog, name) subdata = { 'name': name if not subalias else '%s (%s)' % (name, ', '.join(subalias)), 'help': helps.get(name, ''), 'usage': subaction.format_usage().strip(), 'bare_usage': _format_usage_without_prefix(subaction), } if subalias: subdata['identifier'] = name parse_parser(subaction, subdata, **kwargs) data.setdefault('children', []).append(subdata) continue if 'args' not in data: data['args'] = [] arg = { 'name': action.dest, 'help': action.help or '', 'metavar': action.metavar } if action.choices: arg['choices'] = action.choices data['args'].append(arg) show_defaults = ( ('skip_default_values' not in kwargs) or (kwargs['skip_default_values'] is False)) for action in parser._get_optional_actions(): if isinstance(action, _HelpAction): continue if 'options' not in data: data['options'] = [] option = { 'name': action.option_strings, 'default': action.default if show_defaults else '==SUPPRESS==', 'help': action.help or '' } if action.choices: option['choices'] = action.choices if "==SUPPRESS==" not in option['help']: data['options'].append(option) return data package-manager-3.0.1/doc/ext/sphinxarg/LICENSE0000644000175000017500000000206714565163601017215 0ustar rharhaThe MIT License (MIT) Copyright (c) 2013 Alex Rudakov Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. package-manager-3.0.1/doc/ext/sphinxarg/ext.py0000644000175000017500000004170214565163601017361 0ustar rharhafrom argparse import ArgumentParser import os from docutils import nodes from docutils.statemachine import StringList from docutils.parsers.rst.directives import flag, unchanged try: # Removed as of Sphinx 1.7 from sphinx.util.compat import Directive except ImportError as err: from docutils.parsers.rst import Directive from sphinx.util.nodes import nested_parse_with_titles from sphinxarg.parser import parse_parser, parser_navigate def map_nested_definitions(nested_content): if nested_content is None: raise Exception('Nested content should be iterable, not null') # build definition dictionary definitions = {} for item in nested_content: if not isinstance(item, nodes.definition_list): continue for subitem in item: if not isinstance(subitem, nodes.definition_list_item): continue if not len(subitem.children) > 0: continue classifier = '@after' idx = subitem.first_child_matching_class(nodes.classifier) if idx is not None: ci = subitem[idx] if len(ci.children) > 0: classifier = ci.children[0].astext() if classifier is not None and classifier not in ( '@replace', '@before', '@after'): raise Exception('Unknown classifier: %s' % classifier) idx = subitem.first_child_matching_class(nodes.term) if idx is not None: ch = subitem[idx] if len(ch.children) > 0: term = ch.children[0].astext() idx = subitem.first_child_matching_class(nodes.definition) if idx is not None: def_node = subitem[idx] def_node.attributes['classifier'] = classifier definitions[term] = def_node return definitions def print_arg_list(data, nested_content): definitions = map_nested_definitions(nested_content) items = [] if 'args' in data: for arg in data['args']: my_def = [nodes.paragraph(text=arg['help'])] if arg['help'] else [] name = arg['name'] my_def = apply_definition(definitions, my_def, name) if len(my_def) == 0: my_def.append(nodes.paragraph(text='Undocumented')) if 'choices' in arg: my_def.append(nodes.paragraph( text=('Possible choices: %s' % ', '.join([str(c) for c in arg['choices']])))) argname = name if arg['metavar']: argname = arg['metavar'] items.append( nodes.option_list_item( '', nodes.option_group('', nodes.option('', nodes.option_string(text=argname))), nodes.description('', *my_def))) return nodes.option_list('', *items) if items else None def print_opt_list(data, nested_content): definitions = map_nested_definitions(nested_content) items = [] if 'options' in data: for opt in data['options']: names = [] my_def = [nodes.paragraph(text=opt['help'])] if opt['help'] else [] for name in opt['name']: option_declaration = [nodes.option_string(text=name)] if opt['default'] is not None \ and opt['default'] != '==SUPPRESS==': option_declaration += nodes.option_argument( '', text='=' + str(opt['default'])) names.append(nodes.option('', *option_declaration)) my_def = apply_definition(definitions, my_def, name) if len(my_def) == 0: my_def.append(nodes.paragraph(text='Undocumented')) if 'choices' in opt: my_def.append(nodes.paragraph( text=('Possible choices: %s' % ', '.join([str(c) for c in opt['choices']])))) items.append( nodes.option_list_item( '', nodes.option_group('', *names), nodes.description('', *my_def))) return nodes.option_list('', *items) if items else None def print_command_args_and_opts(arg_list, opt_list, sub_list=None): items = [] if arg_list: items.append(nodes.definition_list_item( '', nodes.term(text='Positional arguments:'), nodes.definition('', arg_list))) if opt_list: items.append(nodes.definition_list_item( '', nodes.term(text='Options:'), nodes.definition('', opt_list))) if sub_list and len(sub_list): items.append(nodes.definition_list_item( '', nodes.term(text='Sub-commands:'), nodes.definition('', sub_list))) return nodes.definition_list('', *items) def apply_definition(definitions, my_def, name): if name in definitions: definition = definitions[name] classifier = definition['classifier'] if classifier == '@replace': return definition.children if classifier == '@after': return my_def + definition.children if classifier == '@before': return definition.children + my_def raise Exception('Unknown classifier: %s' % classifier) return my_def def print_subcommand_list(data, nested_content): definitions = map_nested_definitions(nested_content) items = [] if 'children' in data: for child in data['children']: my_def = [nodes.paragraph( text=child['help'])] if child['help'] else [] name = child['name'] my_def = apply_definition(definitions, my_def, name) if len(my_def) == 0: my_def.append(nodes.paragraph(text='Undocumented')) if 'description' in child: my_def.append(nodes.paragraph(text=child['description'])) my_def.append(nodes.literal_block(text=child['usage'])) my_def.append(print_command_args_and_opts( print_arg_list(child, nested_content), print_opt_list(child, nested_content), print_subcommand_list(child, nested_content) )) items.append( nodes.definition_list_item( '', nodes.term('', '', nodes.strong(text=name)), nodes.definition('', *my_def) ) ) return nodes.definition_list('', *items) class ArgParseDirective(Directive): has_content = True option_spec = dict(module=unchanged, func=unchanged, ref=unchanged, prog=unchanged, path=unchanged, nodefault=flag, manpage=unchanged, nosubcommands=unchanged, passparser=flag) def _construct_manpage_specific_structure(self, parser_info): """ Construct a typical man page consisting of the following elements: NAME (automatically generated, out of our control) SYNOPSIS DESCRIPTION OPTIONS FILES SEE ALSO BUGS """ # SYNOPSIS section synopsis_section = nodes.section( '', nodes.title(text='Synopsis'), nodes.literal_block(text=parser_info["bare_usage"]), ids=['synopsis-section']) # DESCRIPTION section description_section = nodes.section( '', nodes.title(text='Description'), nodes.paragraph(text=parser_info.get( 'description', parser_info.get( 'help', "undocumented").capitalize())), ids=['description-section']) nested_parse_with_titles( self.state, self.content, description_section) if parser_info.get('epilog'): # TODO: do whatever sphinx does to understand ReST inside # docstrings magically imported from other places. The nested # parse method invoked above seem to be able to do this but # I haven't found a way to do it for arbitrary text description_section += nodes.paragraph( text=parser_info['epilog']) # OPTIONS section options_section = nodes.section( '', nodes.title(text='Options'), ids=['options-section']) if 'args' in parser_info: options_section += nodes.paragraph() options_section += nodes.subtitle(text='Positional arguments:') options_section += self._format_positional_arguments(parser_info) if 'options' in parser_info: options_section += nodes.paragraph() options_section += nodes.subtitle(text='Optional arguments:') options_section += self._format_optional_arguments(parser_info) items = [ # NOTE: we cannot generate NAME ourselves. It is generated by # docutils.writers.manpage synopsis_section, description_section, # TODO: files # TODO: see also # TODO: bugs ] if len(options_section.children) > 1: items.append(options_section) if 'nosubcommands' not in self.options: # SUBCOMMANDS section (non-standard) subcommands_section = nodes.section( '', nodes.title(text='Sub-Commands'), ids=['subcommands-section']) if 'children' in parser_info: subcommands_section += self._format_subcommands(parser_info) if len(subcommands_section) > 1: items.append(subcommands_section) if os.getenv("INCLUDE_DEBUG_SECTION"): import json # DEBUG section (non-standard) debug_section = nodes.section( '', nodes.title(text="Argparse + Sphinx Debugging"), nodes.literal_block(text=json.dumps(parser_info, indent=' ')), ids=['debug-section']) items.append(debug_section) return items def _format_positional_arguments(self, parser_info): assert 'args' in parser_info items = [] for arg in parser_info['args']: arg_items = [] if arg['help']: arg_items.append(nodes.paragraph(text=arg['help'])) else: arg_items.append(nodes.paragraph(text='Undocumented')) if 'choices' in arg: arg_items.append( nodes.paragraph( text='Possible choices: ' + ', '.join(arg['choices']))) items.append( nodes.option_list_item( '', nodes.option_group( '', nodes.option( '', nodes.option_string(text=arg['metavar']) ) ), nodes.description('', *arg_items))) return nodes.option_list('', *items) def _format_optional_arguments(self, parser_info): assert 'options' in parser_info items = [] for opt in parser_info['options']: names = [] opt_items = [] for name in opt['name']: option_declaration = [nodes.option_string(text=name)] if opt['default'] is not None \ and opt['default'] != '==SUPPRESS==': option_declaration += nodes.option_argument( '', text='=' + str(opt['default'])) names.append(nodes.option('', *option_declaration)) if opt['help']: opt_items.append(nodes.paragraph(text=opt['help'])) else: opt_items.append(nodes.paragraph(text='Undocumented')) if 'choices' in opt: opt_items.append( nodes.paragraph( text='Possible choices: ' + ', '.join(opt['choices']))) items.append( nodes.option_list_item( '', nodes.option_group('', *names), nodes.description('', *opt_items))) return nodes.option_list('', *items) def _format_subcommands(self, parser_info): assert 'children' in parser_info items = [] for subcmd in parser_info['children']: subcmd_items = [] if subcmd['help']: subcmd_items.append(nodes.paragraph(text=subcmd['help'])) else: subcmd_items.append(nodes.paragraph(text='Undocumented')) items.append( nodes.definition_list_item( '', nodes.term('', '', nodes.strong( text=subcmd['bare_usage'])), nodes.definition('', *subcmd_items))) return nodes.definition_list('', *items) def _nested_parse_paragraph(self, text): content = nodes.paragraph() self.state.nested_parse(StringList(text.split("\n")), 0, content) return content def run(self): if 'module' in self.options and 'func' in self.options: module_name = self.options['module'] attr_name = self.options['func'] elif 'ref' in self.options: _parts = self.options['ref'].split('.') module_name = '.'.join(_parts[0:-1]) attr_name = _parts[-1] else: raise self.error( ':module: and :func: should be specified, or :ref:') mod = __import__(module_name, globals(), locals(), [attr_name]) file_dependency = mod.__file__ if file_dependency.endswith('.pyc'): file_dependency = file_dependency[:-1] env = self.state.document.settings.env if not hasattr(env, 'argparse_usages'): env.argparse_usages = [] env.argparse_usages.append({ 'docname': env.docname, 'lineno': self.lineno, 'dependency_file': file_dependency, 'dependency_mtime': os.stat(file_dependency).st_mtime, }) if not hasattr(mod, attr_name): raise self.error(( 'Module "%s" has no attribute "%s"\n' 'Incorrect argparse :module: or :func: values?' ) % (module_name, attr_name)) func = getattr(mod, attr_name) if isinstance(func, ArgumentParser): parser = func elif 'passparser' in self.options: parser = ArgumentParser() func(parser) else: parser = func() if 'path' not in self.options: self.options['path'] = '' path = str(self.options['path']) if 'prog' in self.options: parser.prog = self.options['prog'] result = parse_parser( parser, skip_default_values='nodefault' in self.options) result = parser_navigate(result, path) if 'manpage' in self.options: return self._construct_manpage_specific_structure(result) nested_content = nodes.paragraph() self.state.nested_parse( self.content, self.content_offset, nested_content) nested_content = nested_content.children items = [] # add common content between for item in nested_content: if not isinstance(item, nodes.definition_list): items.append(item) if 'description' in result: items.append(self._nested_parse_paragraph(result['description'])) items.append(nodes.literal_block(text=result['usage'])) if 'nosubcommands' in self.options: subcommands = None else: subcommands = print_subcommand_list(result, nested_content) items.append(print_command_args_and_opts( print_arg_list(result, nested_content), print_opt_list(result, nested_content), subcommands )) if 'epilog' in result: items.append(self._nested_parse_paragraph(result['epilog'])) return items def env_get_outdated_hook(app, env, added, changed, removed): from sphinx.util import logging logger = logging.getLogger(__name__) rval = set() if not hasattr(env, 'argparse_usages'): return [] for usage in env.argparse_usages: docname = usage['docname'] dep_file = usage['dependency_file'] dep_mtime = usage['dependency_mtime'] current_mtime = os.stat(dep_file).st_mtime if current_mtime > dep_mtime and docname not in removed: rval.add(docname) for docname in rval: from sphinx.util.console import blue msg = blue('found outdated argparse doc: {0}'.format(docname)) logger.info(msg) return list(rval) def env_purge_doc_hook(app, env, docname): if not hasattr(env, 'argparse_usages'): return env.argparse_usages = [ usage for usage in env.argparse_usages if usage['docname'] != docname] def setup(app): app.add_directive('argparse', ArgParseDirective) app.connect('env-get-outdated', env_get_outdated_hook) app.connect('env-purge-doc', env_purge_doc_hook) package-manager-3.0.1/doc/ext/sphinxarg/__init__.py0000644000175000017500000000000014565163601020302 0ustar rharhapackage-manager-3.0.1/doc/ext/sphinxarg/README0000644000175000017500000000041214565163601017060 0ustar rharhaModified version of sphinx-argparse 0.1.15: https://github.com/ribozz/sphinx-argparse Added Sphinx extension hooks to check for whether the python module associated with an argparse directive is outdated and so the .rst file it is used in needs to be re-read. package-manager-3.0.1/doc/conf.py0000644000175000017500000002325314565163601014704 0ustar rharha# # Zeek Package Manager documentation build configuration file, created by # sphinx-quickstart on Fri Jul 15 13:46:04 2016. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # import os import sys sys.path.insert(0, os.path.abspath("..")) sys.path.insert(0, os.path.abspath("./ext")) # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. # # needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = ["sphinxarg.ext", "sphinx.ext.autodoc", "sphinx.ext.napoleon"] # Add any paths that contain templates here, relative to this directory. templates_path = ["_templates"] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # # source_suffix = ['.rst', '.md'] source_suffix = ".rst" # The encoding of source files. # # source_encoding = 'utf-8-sig' # The master toctree document. master_doc = "index" # General information about the project. project = "Zeek Package Manager" copyright = "2019, The Zeek Project" author = "The Zeek Project" # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. with open("../VERSION") as f: version = f.readline().strip() # The full version, including alpha/beta/rc tags. release = version # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = "en" # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: # # today = '' # # Else, today_fmt is used as the format for a strftime call. # # today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"] # The reST default role (used for this markup: `text`) to use for all # documents. # # default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. # # add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). # # add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. # # show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = "sphinx" # A list of ignored prefixes for module index sorting. # modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. # keep_warnings = False # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = False # -- Options for HTML output ---------------------------------------------- html_theme = "sphinx_rtd_theme" html_theme_options = { "collapse_navigation": False, "display_version": True, } # The name for this set of Sphinx documents. # " v documentation" by default. # html_title = "Zeek Package Manager Documentation" # A shorter title for the navigation bar. Default is the same as html_title. # # html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. # # html_logo = None # The name of an image file (relative to this directory) to use as a favicon of # the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. # # html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ["_static"] def setup(app): # app.add_javascript("custom.js") app.add_css_file("theme_overrides.css") # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. # # html_extra_path = [] # If not None, a 'Last updated on:' timestamp is inserted at every page # bottom, using the given strftime format. # The empty string is equivalent to '%b %d, %Y'. # # html_last_updated_fmt = None # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. # # html_use_smartypants = True # Custom sidebar templates, maps document names to template names. # # html_sidebars = {'**': ['custom-sidebar.html']} # Additional templates that should be rendered to pages, maps page names to # template names. # # html_additional_pages = {} # If false, no module index is generated. # # html_domain_indices = True # If false, no index is generated. # # html_use_index = True # If true, the index is split into individual pages for each letter. # # html_split_index = False # If true, links to the reST sources are added to the pages. # # html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. # # html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. # # html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. # # html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). # html_file_suffix = None # Language to be used for generating the HTML full-text search index. # Sphinx supports the following languages: # 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' # 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr', 'zh' # # html_search_language = 'en' # A dictionary with options for the search language support, empty by default. # 'ja' uses this config value. # 'zh' user can custom change `jieba` dictionary path. # # html_search_options = {'type': 'default'} # The name of a javascript file (relative to the configuration directory) that # implements a search results scorer. If empty, the default will be used. # # html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. htmlhelp_basename = "ZeekPackageManagerdoc" # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). # # 'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). # # 'pointsize': '10pt', # Additional stuff for the LaTeX preamble. # # 'preamble': '', # Latex figure (float) alignment # # 'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ ( master_doc, "ZeekPackageManager.tex", "Zeek Package Manager Documentation", "The Zeek Project", "manual", ), ] # The name of an image file (relative to this directory) to place at the top of # the title page. # # latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. # # latex_use_parts = False # If true, show page references after internal links. # # latex_show_pagerefs = False # If true, show URL addresses after external links. # # latex_show_urls = False # Documents to append as an appendix to all manuals. # # latex_appendices = [] # It false, will not define \strong, \code, itleref, \crossref ... but only # \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added # packages. # # latex_keep_old_macro_names = True # If false, no module index is generated. # # latex_domain_indices = True # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [("zkg", "zkg", "Zeek Package Manager", [author], 1)] # If true, show URL addresses after external links. # # man_show_urls = False # -- Options for Texinfo output ------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ ( master_doc, "ZeekPackageManager", "Zeek Package Manager Documentation", author, "ZeekPackageManager", "One line description of project.", "Miscellaneous", ), ] # Documents to append as an appendix to all manuals. # # texinfo_appendices = [] # If false, no module index is generated. # # texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. # # texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. # # texinfo_no_detailmenu = False package-manager-3.0.1/doc/quickstart.rst0000644000175000017500000002271514565163601016333 0ustar rharha.. _PyPI: https://pypi.python.org/pypi .. _ZeekControl: https://github.com/zeek/zeekctl Quickstart Guide ================ Dependencies ------------ * Python 3.6+ * git: https://git-scm.com * GitPython: https://pypi.python.org/pypi/GitPython * semantic_version: https://pypi.python.org/pypi/semantic_version * btest: https://pypi.python.org/pypi/btest Note that following the :program:`zkg` `Installation`_ process via :program:`pip3` will automatically install its dependencies for you. Installation ------------ * Zeek 4.0.0 or greater comes with a bundled :program:`zkg` that is included as part of its installation. This is often the easiest choice since it comes pre-configured to work correctly for that particular Zeek installation and some `Basic Configuration`_ steps can be skipped. The directions to build and install Zeek from source are here: https://docs.zeek.org/en/current/install/install.html Note that this method does require independent installation of :program:`zkg`'s dependencies, which is usually easiest to do via :program:`pip3`: .. code-block:: console $ pip3 install gitpython semantic-version * To install the latest release of :program:`zkg` on PyPI_: .. code-block:: console $ pip3 install zkg * To install the latest Git development version of :program:`zkg`: .. code-block:: console $ pip3 install git+git://github.com/zeek/package-manager@master .. note:: If not using something like :program:`virtualenv` to manage Python environments, the default user script directory is :file:`~/.local/bin` and you may have to modify your :envvar:`PATH` to search there for :program:`zkg`. Basic Configuration ------------------- :program:`zkg` supports four broad approaches to managing Zeek packages: - Keep package metadata in :file:`$HOME/.zkg/` and maintain Zeek-relevant package content (such as scripts and plugins) in the Zeek installation tree. This is :program:`zkg`'s "traditional" approach. - Keep all state and package content within the Zeek installation tree. Zeek 4's bundled :program:`zkg` installation provides this by default. If you use multiple Zeek installations in parallel, this approach allows you to install different sets of Zeek packages with each Zeek version. - Keep all state and package content in :file:`$HOME/.zkg/`. This is the preferred approach when you're running :program:`zkg` and :program:`zeek` as different users. :program:`zkg`'s ``--user`` mode enables this approach. - Custom configurations where you select your own state and content locations. After installing via :program:`pip3`, but not when using the :program:`zkg` that comes pre-bundled with a Zeek installation, additional configuration is still required in the form of running a ``zkg autoconfig`` command, but in either case, do read onward to get a better understanding of how the package manager is configured, what directories it uses, etc. To configure :program:`zkg` for use with a given Zeek installation, make sure that the :program:`zeek-config` script that gets installed with :program:`zeek` is in your :envvar:`PATH`. Then, as the user you want to run :program:`zkg` with, do: .. code-block:: console $ zkg autoconfig This automatically generates a config file with the following suggested settings that should work for most Zeek deployments: - `script_dir`: set to the location of Zeek's :file:`site` scripts directory (e.g. :file:`{}/share/zeek/site`) - `plugin_dir`: set to the location of Zeek's default plugin directory (e.g. :file:`{}/lib/zeek/plugins`) - `bin_dir`: set to the location where :program:`zkg` installs executables that packages provide (e.g., :file:`{}/bin`). - `zeek_dist`: set to the location of Zeek's source code. If you didn't build/install Zeek from source code, this field will not be set, but it's only needed if you plan on installing packages that have uncompiled Zeek plugins. With those settings, the package manager will install Zeek scripts, Zeek plugins, and ZeekControl plugins into directories where :program:`zeek` and :program:`zeekctl` will, by default, look for them. ZeekControl clusters will also automatically distribute installed package scripts/plugins to all nodes. .. note:: If your Zeek installation is owned by "root" and you intend to run :program:`zkg` as a different user, you have two options. First, you can use :program:`zkg`'s user mode (``zkg --user``). In user mode, :program:`zkg` consults :file:`$HOME/.zkg/config` for configuration settings. Creating this config file in user mode (``zkg --user autoconfig``) ensures that all state and content directories reside within :file:`$HOME/.zkg/`. :program:`zkg` reports according environment variables in the output of ``zkg --user env``. Second, you can grant "write" access to the directories specified by `script_dir`, `plugin_dir`, and `bin_dir`; perhaps using something like: .. code-block:: console $ sudo chgrp $USER $(zeek-config --site_dir) $(zeek-config --plugin_dir) $(zeek-config --prefix)/bin $ sudo chmod g+rwX $(zeek-config --site_dir) $(zeek-config --plugin_dir) $(zeek-config --prefix)/bin The final step is to edit your :file:`site/local.zeek`. If you want to have Zeek automatically load the scripts from all :ref:`installed ` packages that are also marked as ":ref:`loaded `" add: .. code-block:: zeek @load packages If you prefer to manually pick the package scripts to load, you may instead add lines like :samp:`@load {}`, where :samp:`{}` is the :ref:`shorthand name ` of the desired package. If you want to further customize your configuration, see the `Advanced Configuration`_ section and also check :ref:`here ` for a full explanation of config file options. Otherwise you're ready to use :ref:`zkg `. Advanced Configuration ---------------------- If you prefer to not use the suggested `Basic Configuration`_ settings for `script_dir` and `plugin_dir`, the default configuration will install all package scripts/plugins within :file:`~/.zkg` or you may change them to whatever location you prefer. These will be referred to as "non-standard" locations in the sense that vanilla configurations of either :program:`zeek` or :program:`zeekctl` will not detect scripts/plugins in those locations without additional configuration. When using non-standard location, follow these steps to integrate with :program:`zeek` and :program:`zeekctl`: - To get command-line :program:`zeek` to be aware of Zeek scripts/plugins in a non-standard location, make sure the :program:`zeek-config` script (that gets installed along with :program:`zeek`) is in your :envvar:`PATH` and run: .. code-block:: console $ `zkg env` Note that this sets up the environment only for the current shell session. - To get :program:`zeekctl` to be aware of scripts/plugins in a non-standard location, run: .. code-block:: console $ zkg config script_dir And set the `SitePolicyPath` option in :file:`zeekctl.cfg` based on the output you see. Similarly, run: .. code-block:: console $ zkg config plugin_dir And set the `SitePluginPath` option in :file:`zeekctl.cfg` based on the output you see. - To have your shell find executables that packages provide, update your :envvar:`PATH`: .. code-block:: console $ export PATH=$(zkg config bin_dir):$PATH (Executing ```zkg env```, as described above, includes this already.) Usage ----- Check the output of :ref:`zkg --help ` for an explanation of all available functionality of the command-line tool. Package Upgrades/Versioning ~~~~~~~~~~~~~~~~~~~~~~~~~~~ When installing packages, note that the :ref:`install command `, has a ``--version`` flag that may be used to install specific package versions which may either be git release tags or branch names. The way that :program:`zkg` receives updates for a package depends on whether the package is first installed to track stable releases or a specific git branch. See the :ref:`package upgrade process ` documentation to learn how :program:`zkg` treats each situation. Offline Usage ~~~~~~~~~~~~~ It's common to have limited network/internet access on the systems where Zeek is deployed. To accomodate those scenarios, :program:`zkg` can be used as normally on a system that *does* have network access to create bundles of its package installation environment. Those bundles can then be transferred to the deployment systems via whatever means are appropriate (SSH, USB flash drive, etc). For example, on the package management system you can do typical package management tasks, like install and update packages: .. code-block:: console $ zkg install Then, via the :ref:`bundle command `, create a bundle file which contains a snapshot of all currently installed packages: .. code-block:: console $ zkg bundle zeek-packages.bundle Then transfer :file:`zeek-packages.bundle` to the Zeek deployment management host. For Zeek clusters using ZeekControl_, this will be the system acting as the "manager" node. Then on that system (assuming it already as :program:`zkg` installed and configured): .. code-block:: console $ zkg unbundle zeek-packages.bundle Finally, if you're using ZeekControl_, and the unbundling process was successful, you need to deploy the changes to worker nodes: .. code-block:: console $ zeekctl deploy package-manager-3.0.1/doc/_static/0000755000175000017500000000000014565163601015026 5ustar rharhapackage-manager-3.0.1/doc/_static/theme_overrides.css0000644000175000017500000000057614565163601020734 0ustar rharha/* Prevent sphinx-argparse option output (e.g. --version) from wrapping. */ span.option { white-space: nowrap; } /* override table width restrictions */ html body div.wy-table-responsive table td, html body div.wy-table-responsive table th { white-space: normal; } html body div.wy-table-responsive { margin-bottom: 24px; max-width: 100%; overflow: visible; } package-manager-3.0.1/doc/_templates/0000755000175000017500000000000014565163601015535 5ustar rharhapackage-manager-3.0.1/doc/_templates/layout.html0000644000175000017500000000035614565163601017744 0ustar rharha{% extends "!layout.html" %} {% if READTHEDOCS and current_version %} {% if current_version == "latest" or current_version == "stable" %} {% set current_version = current_version ~ " (" ~ version ~ ")" %} {% endif %} {% endif %} package-manager-3.0.1/doc/_templates/breadcrumbs.html0000644000175000017500000000122014565163601020707 0ustar rharha{% extends "!breadcrumbs.html" %} {% block breadcrumbs_aside %}
  • {% if pagename != "search" %} {% if display_github %} {% if github_version == "master" %} {{ _('Edit on GitHub') }} {% endif %} {% elif show_source and has_source and sourcename %} {{ _('View page source') }} {% endif %} {% endif %}
    • {% endblock %} package-manager-3.0.1/doc/Makefile0000644000175000017500000001740014565163601015042 0ustar rharha# Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = _build # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " applehelp to make an Apple Help Book" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " epub3 to make an epub3" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " coverage to run coverage check of the documentation (if enabled)" @echo " dummy to check syntax errors of document sources" .PHONY: clean clean: rm -rf $(BUILDDIR)/* .PHONY: html html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html touch $(BUILDDIR)/html/.nojekyll @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." .PHONY: livehtml livehtml: sphinx-autobuild --ignore "../testing/*" --ignore "*.git/*" --ignore "*.lock" --ignore "*.pyc" --ignore "*.swp" --ignore "*.swpx" --ignore "*.swx" --watch .. --watch ../zeekpkg -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html .PHONY: dirhtml dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." .PHONY: singlehtml singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." .PHONY: pickle pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." .PHONY: json json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." .PHONY: htmlhelp htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." .PHONY: qthelp qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/ZeekPackageManager.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/ZeekPackageManager.qhc" .PHONY: applehelp applehelp: $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp @echo @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." @echo "N.B. You won't be able to view it unless you put it in" \ "~/Library/Documentation/Help or install it in your application" \ "bundle." .PHONY: devhelp devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/ZeekPackageManager" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/ZeekPackageManager" @echo "# devhelp" .PHONY: epub epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." .PHONY: epub3 epub3: $(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3 @echo @echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3." .PHONY: latex latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." .PHONY: latexpdf latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." .PHONY: latexpdfja latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." .PHONY: text text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." .PHONY: man man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." .PHONY: texinfo texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." .PHONY: info info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." .PHONY: gettext gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." .PHONY: changes changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." .PHONY: linkcheck linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." .PHONY: doctest doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." .PHONY: coverage coverage: $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage @echo "Testing of coverage in the sources finished, look at the " \ "results in $(BUILDDIR)/coverage/python.txt." .PHONY: xml xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." .PHONY: pseudoxml pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." .PHONY: dummy dummy: $(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy @echo @echo "Build finished. Dummy builder generates no files." package-manager-3.0.1/doc/developers.rst0000644000175000017500000002102614565163601016303 0ustar rharha.. _Sphinx: http://www.sphinx-doc.org .. _Read the Docs: https://docs.zeek.org/projects/package-manager .. _GitHub: https://github.com/zeek/package-manager .. _Google Style Docstrings: http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html .. _zeek-aux: https://github.com/zeek/zeek-aux .. _PyPi: https://pypi.python.org/pypi Developer's Guide ================= This a guide for developers working on the Zeek Package Manager itself. Versioning/Releases ------------------- After making a commit to the *master* branch, you can use the :program:`update-changes` script in the `zeek-aux`_ repository to automatically adapt version numbers and regenerate the :program:`zkg` man page. Make sure to install the `documentation dependencies`_ before using it. Releases are hosted at PyPi_. To build and upload a release: #. Finalize the git repo tag and version with ``update-changes -R `` if not done already. #. Upload the distribution (you will need the credentials for the 'zeek' account on PyPi): .. code-block:: console $ make upload Documentation ------------- Documentation is written in reStructuredText (reST), which Sphinx_ uses to generate HTML documentation and a man page. .. _documentation dependencies: Dependencies ~~~~~~~~~~~~ To build documentation locally, find the requirements in :file:`requirements.txt`: .. literalinclude:: ../requirements.txt They can be installed like:: pip3 install -r requirements.txt Local Build/Preview ~~~~~~~~~~~~~~~~~~~ Use the :file:`Makefile` targets ``make html`` and ``make man`` to build the HTML and man page, respectively, or ``make doc`` to build them both. To view the generated HTML output, open :file:`doc/_build/index.html`. The generated man page is located in :file:`doc/man/zkg.1`. If you have also installed :program:`sphinx-autobuild` (e.g. via :program:`pip3`), there's a :file:`Makefile` target, ``make livehtml``, you can use to help preview documentation changes as you edit the reST files. Remote Hosting ~~~~~~~~~~~~~~ The GitHub_ repository has a webhook configured to automatically rebuild the HTML documentation hosted at `Read the Docs`_ whenever a commit is pushed. Style Conventions ~~~~~~~~~~~~~~~~~ The following style conventions are (generally) used. ========================== =============================== =========================== Documentation Subject reST Markup Preview ========================== =============================== =========================== File Path ``:file:`path``` :file:`path` File Path w/ Substitution ``:file:`{}/path``` :file:`{}/path` OS-Level Commands ``:command:`cmd``` :command:`cmd` Program Names ``:program:`prog``` :program:`prog` Environment Variables ``:envvar:`VAR``` :envvar:`VAR` Literal Text (e.g. code) ````code```` ``code`` Substituted Literal Text ``:samp:`code {}``` :samp:`code {}` Variable/Type Name ```x``` `x` INI File Option ```name``` `name` ========================== =============================== =========================== Python API docstrings roughly follow the `Google Style Docstrings`_ format. Internals --------- ``zkg``'s view of a package ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``zkg`` maintains copies of a Zeek package in up to four places: - A long-lived clone of the package's git repo in ``$STATEDIR/clones/package/``. This clone (not its installed version, see below) is ``zkg``'s "authoritative" view of a package. - A conceptually short-lived clone in ``$STATEDIR/scratch/``, for retrieving information about a package ("short-lived", because access to those copies is momentary -- not because ``zkg`` necessarily cleans up after those copies). - A "stage". A stage is any place in which ``zkg`` actually installs a package for running in Zeek. This can be the local Zeek installation, or locally below ``$STATEDIR/testing`` when testing a package. So "staging" here does not mean "not yet live"; rather the opposite: it's closer to "any place where the package may actually run". A stage allows customization of many of the directories involved (for internal cloning, installing Zeek scripts and plugins, binary files, etc). Installation into a stage is also where ``zkg`` adds its management of ``packages.zeek``, in the stage's scripts directory. - In ``$STATEDIR/testing//clones/``, for testing a given package along with any dependencies it might have. ``zkg`` captures state about installed packages in ``$STATEDIR/manifest.json``. This does not capture all knowable information about packages, though. More on this next. Directory usage ~~~~~~~~~~~~~~~ ``zkg`` populates its internal state directory (dubbed ``$STATEDIR`` below) with several subdirectories. ``$STATEDIR/clones`` """""""""""""""""""" This directory keeps git clones of installed packages (``$STATEDIR/clones/package/``), packages sources (``$STATEDIR/clones/source/``), and package template repositories (``$STATEDIR/clones/template/``). ``zkg`` clones the relevant repositories as needed. It can dynamically re-create some of these directories as needed, but interfering in that space is not recommended. For example, if you remove a clone of an installed package, the installation itself will remain live (via the staging mechanism), but ``zkg`` will no longer be able to refer to all information about the installed package (because anything not explicitly captured about the package in ``$STATEDIR/manifest.json`` is now gone). Removal of a package (``zkg remove``) removes its clone in ``$STATEDIR/clones``. ``$STATEDIR/scratch`` """"""""""""""""""""" When retrieving information on a package that isn't yet installed, or where ``zkg`` doesn't want to touch the installed code, ``$STATEDIR/scratch/`` is a clone of the package's git repo at a version (often a git tag) of interest. This clone is shallow for any versions that aren't raw SHA-1 hashes. The information parsed includes the ``zkg.meta`` as well as git branch/tag/commit info. During package installation, ``zkg`` places backups of user-tweakable files into ``$STATEDIR/scratch/tmpcfg``. ``zkg`` restores these after package installation to preserve the user's edits. During package source aggregation, ``zkg`` places temporary versions of ``aggregate.meta`` directly into ``$STATEDIR/scratch``. Creation or unbundling of a package happens via ``$STATEDIR/scratch/bundle``, to compose or retrieve information about the bundle. The directory is deleted and re-created at the beginning of those operations: - During bundling, ``zkg`` copies installed package repos from ``$STATEDIR/clones/`` into ``$STATEDIR/scratch/bundle/``, and creates fresh git clones in the ``bundle`` directory for any packages not currently installed. It creates a ``.tar.gz`` of the whole directory, initially in the bundle directory, and moves it to where the user specified. - During unbundling, ``zkg`` reads the bundle manifest as well as the git repos of the contained packages, and moves the package repos from the scratch space into ``$STATEDIR/clones/package/``. When installing a package's ``script_dir`` or ``plugin_dir`` into a staging area and the source file is a tarfile, ``zkg`` temporarily extracts the tarball into ``$STATEDIR/scratch/untar/``. There's little or no cleanup of files in the scratch space after the operations creating them complete. ``zkg`` only deletes, then (re-)creates, the directories involved upon next use. When ``zkg`` isn't in the middle of executing any commands, you can always delete the scratch space without negatively affecting ``zkg``. ``$STATEDIR/testing`` """"""""""""""""""""" When testing a package (during installation, or when explicitly running ``zkg test``), ``zkg`` creates a staging area ``$STATEDIR/testing/`` for the package under test, clones the package and its dependencies into ``$STATEDIR/testing//clones/``, installs them from there into ``$STATEDIR/testing/``, and then runs the package's ``test_command`` from its clone (``$STATEDIR/testing//clones/``), with an environment set such that it finds the installation in the local stage. The stdout and stderr of those testing runs is preserved into ``zkg.test_command.stdout`` and ``zkg.test_command.stderr`` in that directory. As in ``STATEDIR/scratch``, there's no cleanup, and you can delete the testing space as needed after a test run is complete. package-manager-3.0.1/.pre-commit-config.yaml0000644000175000017500000000124614565163601017117 0ustar rharha# See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v4.4.0 hooks: - id: trailing-whitespace exclude: testing/baselines - id: end-of-file-fixer exclude: testing/baselines - id: check-yaml - id: check-added-large-files - repo: https://github.com/psf/black rev: 23.1a1 hooks: - id: black - repo: https://github.com/asottile/pyupgrade rev: v3.3.1 hooks: - id: pyupgrade args: ["--py37-plus"] - repo: https://github.com/pycqa/flake8 rev: 5.0.4 # 6.0.0 requires Python 3.8 hooks: - id: flake8 exclude: doc/ext/sphinxarg package-manager-3.0.1/zkg0000755000175000017500000031134614565163601013364 0ustar rharha#! /usr/bin/env python3 # https://pypi.org/project/argcomplete/#global-completion # PYTHON_ARGCOMPLETE_OK import argparse import configparser import errno import filecmp import io import json import logging import os import shutil import subprocess import sys import threading from collections import OrderedDict try: import git import semantic_version # noqa # pylint: disable=unused-import except ImportError as error: print( "error: zkg failed to import one or more dependencies:\n" "\n" "* GitPython: https://pypi.org/project/GitPython\n" "* semantic-version: https://pypi.org/project/semantic-version\n" "\n" "If you use 'pip', they can be installed like:\n" "\n" " pip3 install GitPython semantic-version\n" "\n" "Also check the following exception output for possible alternate explanations:\n\n" "{}: {}".format(type(error).__name__, error), file=sys.stderr, ) sys.exit(1) try: # Argcomplete provides command-line completion for users of argparse. # We support it if available, but don't complain when it isn't. import argcomplete # pylint: disable=import-error except ImportError: pass # For Zeek-bundled installation, prepend the Python path of the Zeek # installation. This ensures we find the matching zeekpkg module # first, avoiding potential conflicts with installations elsewhere on # the system. ZEEK_PYTHON_DIR = "@PY_MOD_INSTALL_DIR@" if os.path.isdir(ZEEK_PYTHON_DIR): sys.path.insert(0, os.path.abspath(ZEEK_PYTHON_DIR)) else: ZEEK_PYTHON_DIR = None # Similarly, make Zeek's binary installation path available by # default. This helps package installations succeed that require # e.g. zeek-config for their build process. ZEEK_BIN_DIR = "@ZEEK_BIN_DIR@" if os.path.isdir(ZEEK_BIN_DIR): try: if ZEEK_BIN_DIR not in os.environ["PATH"].split(os.pathsep): os.environ["PATH"] = ZEEK_BIN_DIR + os.pathsep + os.environ["PATH"] except KeyError: os.environ["PATH"] = ZEEK_BIN_DIR else: ZEEK_BIN_DIR = None # Also when bundling with Zeek, use directories in the install tree # for storing the zkg configuration and its variable state. Support # for overrides via environment variables simplifies testing. ZEEK_ZKG_CONFIG_DIR = os.getenv("ZEEK_ZKG_CONFIG_DIR") or "@ZEEK_ZKG_CONFIG_DIR@" if not os.path.isdir(ZEEK_ZKG_CONFIG_DIR): ZEEK_ZKG_CONFIG_DIR = None ZEEK_ZKG_STATE_DIR = os.getenv("ZEEK_ZKG_STATE_DIR") or "@ZEEK_ZKG_STATE_DIR@" if not os.path.isdir(ZEEK_ZKG_STATE_DIR): ZEEK_ZKG_STATE_DIR = None # The default package source we fall back to as needed ZKG_DEFAULT_SOURCE = "https://github.com/zeek/packages" # The default package template ZKG_DEFAULT_TEMPLATE = "https://github.com/zeek/package-template" from zeekpkg._util import ( delete_path, make_dir, find_program, read_zeek_config_line, std_encoding, ) from zeekpkg.package import ( BUILTIN_SCHEME, TRACKING_METHOD_VERSION, ) from zeekpkg.template import ( LoadError, Template, ) from zeekpkg.uservar import ( UserVar, ) import zeekpkg def confirmation_prompt(prompt, default_to_yes=True): yes = {"y", "ye", "yes"} if default_to_yes: prompt += " [Y/n] " else: prompt += " [N/y] " choice = input(prompt).lower() if not choice: if default_to_yes: return True else: print("Abort.") return False if choice in yes: return True print("Abort.") return False def prompt_for_user_vars(manager, config, configfile, args, pkg_infos): answers = {} for info in pkg_infos: name = info.package.qualified_name() requested_user_vars = info.user_vars() if requested_user_vars is None: print_error(str.format('error: malformed user_vars in "{}"', name)) sys.exit(1) for uvar in requested_user_vars: try: answers[uvar.name()] = uvar.resolve( name, config, args.user_var, args.force ) except ValueError: print_error( str.format( 'error: could not determine value of user variable "{}",' " provide via environment or --user-var", uvar.name(), ) ) sys.exit(1) if not args.force and answers: for key, value in answers.items(): if not config.has_section("user_vars"): config.add_section("user_vars") config.set("user_vars", key, value) if configfile: with open(configfile, "w", encoding=std_encoding(sys.stdout)) as f: config.write(f) print(f"Saved answers to config file: {configfile}") manager.user_vars = answers def print_error(*args, **kwargs): print(*args, file=sys.stderr, **kwargs) def config_items(config, section): # Same as config.items(section), but exclude default keys. defaults = {key for key, _ in config.items("DEFAULT")} items = sorted(config.items(section)) return [(key, value) for (key, value) in items if key not in defaults] def file_is_not_empty(path): return os.path.isfile(path) and os.path.getsize(path) > 0 def find_configfile(args): if args.user: configfile = os.path.join(home_config_dir(), "config") if file_is_not_empty(configfile): return configfile return None configfile = os.environ.get("ZKG_CONFIG_FILE") if configfile and file_is_not_empty(configfile): return configfile configfile = os.path.join(default_config_dir(), "config") if file_is_not_empty(configfile): return configfile return None def home_config_dir(): return os.path.join(os.path.expanduser("~"), ".zkg") def default_config_dir(): return ZEEK_ZKG_CONFIG_DIR or home_config_dir() def default_state_dir(): return ZEEK_ZKG_STATE_DIR or home_config_dir() def create_config(args, configfile): config = configparser.ConfigParser() if configfile: if not os.path.isfile(configfile): print_error(f'error: invalid config file "{configfile}"') sys.exit(1) config.read(configfile) if not config.has_section("sources"): config.add_section("sources") if not config.has_section("paths"): config.add_section("paths") if not config.has_section("templates"): config.add_section("templates") if not configfile: default = os.getenv("ZKG_DEFAULT_SOURCE", ZKG_DEFAULT_SOURCE) if default: config.set("sources", "zeek", default) if not configfile or not config.has_option("templates", "default"): default = os.getenv("ZKG_DEFAULT_TEMPLATE", ZKG_DEFAULT_TEMPLATE) if default: config.set("templates", "default", default) def config_option_set(config, section, option): return config.has_option(section, option) and config.get(section, option) def get_option(config, section, option, default): if config_option_set(config, section, option): return config.get(section, option) return default if args.user: def_state_dir = home_config_dir() else: def_state_dir = default_state_dir() state_dir = get_option(config, "paths", "state_dir", os.path.join(def_state_dir)) script_dir = get_option( config, "paths", "script_dir", os.path.join(state_dir, "script_dir") ) plugin_dir = get_option( config, "paths", "plugin_dir", os.path.join(state_dir, "plugin_dir") ) bin_dir = get_option(config, "paths", "bin_dir", os.path.join(state_dir, "bin")) zeek_dist = get_option(config, "paths", "zeek_dist", "") config.set("paths", "state_dir", state_dir) config.set("paths", "script_dir", script_dir) config.set("paths", "plugin_dir", plugin_dir) config.set("paths", "bin_dir", bin_dir) config.set("paths", "zeek_dist", zeek_dist) def expand_config_values(config, section): for key, value in config.items(section): value = os.path.expandvars(os.path.expanduser(value)) config.set(section, key, value) expand_config_values(config, "sources") expand_config_values(config, "paths") expand_config_values(config, "templates") for key, value in config.items("paths"): if value and not os.path.isabs(value): print_error( str.format( "error: invalid config file value for key" ' "{}" in section [paths]: "{}" is not' " an absolute path", key, value, ) ) sys.exit(1) return config def active_git_branch(path): try: repo = git.Repo(path) except git.exc.NoSuchPathError: return None if not repo.working_tree_dir: return None try: rval = repo.active_branch except TypeError: # return detached commit rval = repo.head.commit if not rval: return None rval = str(rval) return rval def is_local_git_repo_url(git_url): return git_url.startswith(".") or git_url.startswith("/") def is_local_git_repo(git_url): try: # The Repo class takes a file system path as first arg. This # can fail in two ways: (1) the path doesn't exist or isn't # accessible, (2) it's not the root directory of a git repo. git.Repo(git_url) return True except (git.exc.InvalidGitRepositoryError, git.exc.NoSuchPathError): return False def is_local_git_repo_dirty(git_url): if not is_local_git_repo(git_url): return False try: repo = git.Repo(git_url) except git.exc.NoSuchPathError: return False return repo.is_dirty(untracked_files=True) def check_local_git_repo(git_url): if is_local_git_repo_url(git_url): if not is_local_git_repo(git_url): print_error(f"error: path {git_url} is not a git repository") return False if is_local_git_repo_dirty(git_url): print_error(f"error: local git clone at {git_url} is dirty") return False return True def create_manager(args, config): state_dir = config.get("paths", "state_dir") script_dir = config.get("paths", "script_dir") plugin_dir = config.get("paths", "plugin_dir") bin_dir = config.get("paths", "bin_dir") zeek_dist = config.get("paths", "zeek_dist") try: manager = zeekpkg.Manager( state_dir=state_dir, script_dir=script_dir, plugin_dir=plugin_dir, bin_dir=bin_dir, zeek_dist=zeek_dist, ) except OSError as error: if error.errno == errno.EACCES: print_error(f"{type(error).__name__}: {error}") def check_permission(d): if os.access(d, os.W_OK): return True print_error(f"error: user does not have write access in {d}") return False permissions_trouble = not all( [ check_permission(state_dir), check_permission(script_dir), check_permission(plugin_dir), check_permission(bin_dir), ] ) if permissions_trouble and not args.user: print_error( "Consider the --user flag to manage zkg state via {}/config".format( home_config_dir() ) ) sys.exit(1) raise extra_sources = [] for key_value in args.extra_source or []: if "=" not in key_value: print_error(f'warning: invalid extra source: "{key_value}"') continue key, value = key_value.split("=", 1) if not key or not value: print_error(f'warning: invalid extra source: "{key_value}"') continue extra_sources.append((key, value)) for key, value in extra_sources + config_items(config, "sources"): error = manager.add_source(name=key, git_url=value) if error: print_error( str.format( 'warning: skipped using package source named "{}": {}', key, error ) ) return manager def get_changed_state(manager, saved_state, pkg_lst): """Returns the list of packages that have changed loaded state. Args: saved_state (dict): dictionary of saved load state for installed packages. pkg_lst (list): list of package names to be skipped Returns: dep_listing (str): string installed packages that have changed state """ _lst = [zeekpkg.package.name_from_path(_pkg_path) for _pkg_path in pkg_lst] dep_listing = "" for _pkg_name in sorted(manager.installed_package_dependencies()): if _pkg_name in _lst: continue _ipkg = manager.find_installed_package(_pkg_name) if not _ipkg or _ipkg.status.is_loaded == saved_state[_pkg_name]: continue dep_listing += f" {_pkg_name}\n" return dep_listing class InstallWorker(threading.Thread): def __init__(self, manager, package_name, package_version): super().__init__() self.manager = manager self.package_name = package_name self.package_version = package_version self.error = "" def run(self): self.error = self.manager.install(self.package_name, self.package_version) def wait(self, msg=None, out=sys.stdout, tty_only=True): """Blocks until this thread ends, optionally writing liveness indicators. This never returns until this thread dies (i.e., is_alive() is False). When an output file object is provided, the method also indicates progress by writing a dot character to it once per second. This happens only when the file is a TTY, unless ``tty_only`` is False. When a message is given, it gets written out first, regardless of TTY status. Any output always terminates with a newline. Args: msg (str): a message to write first. out (file-like object): the destination to write to. tty_only (bool): whether to write progress dots also to non-TTYs. """ if out is not None and msg: out.write(msg) out.flush() is_tty = hasattr(out, "isatty") and out.isatty() while True: self.join(1.0) if not self.is_alive(): break if out is not None and (is_tty or not tty_only): out.write(".") out.flush() if out is not None and (msg or is_tty or not tty_only): out.write("\n") out.flush() def cmd_test(manager, args, config, configfile): if args.version and len(args.package) > 1: print_error('error: "test --version" may only be used for a single package') sys.exit(1) package_infos = [] for name in args.package: if not check_local_git_repo(name): sys.exit(1) # If the package to be tested is included with Zeek, don't allow # to run tests due to the potential of conflicts. bpkg_info = manager.find_builtin_package(name) if bpkg_info is not None: print_error(f'cannot run tests for "{name}": built-in package') sys.exit(1) version = args.version if args.version else active_git_branch(name) package_info = manager.info(name, version=version, prefer_installed=False) if package_info.invalid_reason: print_error( str.format( 'error: invalid package "{}": {}', name, package_info.invalid_reason ) ) sys.exit(1) if not version: version = package_info.best_version() package_infos.append((package_info, version)) all_passed = True for info, version in package_infos: name = info.package.qualified_name() if "test_command" not in info.metadata: print(str.format("{}: no test_command found in metadata, skipping", name)) continue error_msg, passed, test_dir = manager.test( name, version, test_dependencies=True ) if error_msg: all_passed = False print_error( str.format('error: failed to run tests for "{}": {}', name, error_msg) ) continue if passed: print(str.format("{}: all tests passed", name)) else: all_passed = False clone_dir = os.path.join( os.path.join(test_dir, "clones"), info.package.name ) print_error( str.format( 'error: package "{}" tests failed, inspect' " contents of {} for details, especially" ' any "zkg.test_command.{{stderr,stdout}}"' " files within {}", name, test_dir, clone_dir, ) ) if not all_passed: sys.exit(1) def cmd_install(manager, args, config, configfile): if args.version and len(args.package) > 1: print_error('error: "install --version" may only be used for a single package') sys.exit(1) package_infos = [] for name in args.package: if not check_local_git_repo(name): sys.exit(1) # Outright prevent installing a package that Zeek has built-in. bpkg_info = manager.find_builtin_package(name) if bpkg_info is not None: print_error(f'cannot install "{name}": built-in package') sys.exit(1) version = args.version if args.version else active_git_branch(name) package_info = manager.info(name, version=version, prefer_installed=False) if package_info.invalid_reason: print_error( str.format( 'error: invalid package "{}": {}', name, package_info.invalid_reason ) ) sys.exit(1) if not version: version = package_info.best_version() package_infos.append((package_info, version, False)) # We modify package_infos below, so copy current state to preserve list of # packages requested by caller: orig_pkgs, new_pkgs = package_infos.copy(), [] if not args.nodeps: to_validate = [ (info.package.qualified_name(), version) for info, version, _ in package_infos ] invalid_reason, new_pkgs = manager.validate_dependencies( to_validate, ignore_suggestions=args.nosuggestions ) if invalid_reason: print_error("error: failed to resolve dependencies:", invalid_reason) sys.exit(1) if not args.force: package_listing = "" for info, version, _ in sorted(package_infos, key=lambda x: x[0].package.name): name = info.package.qualified_name() package_listing += f" {name} ({version})\n" print("The following packages will be INSTALLED:") print(package_listing) if new_pkgs: dependency_listing = "" for info, version, suggested in sorted( new_pkgs, key=lambda x: x[0].package.name ): name = info.package.qualified_name() dependency_listing += f" {name} ({version})" if suggested: dependency_listing += " (suggested)" dependency_listing += "\n" print("The following dependencies will be INSTALLED:") print(dependency_listing) allpkgs = package_infos + new_pkgs extdep_listing = "" for info, version, _ in sorted(allpkgs, key=lambda x: x[0].package.name): name = info.package.qualified_name() extdeps = info.dependencies(field="external_depends") if extdeps is None: extdep_listing += " from {} ({}):\n \n".format( name, version ) continue if extdeps: extdep_listing += f" from {name} ({version}):\n" for extdep, semver in sorted(extdeps.items()): extdep_listing += f" {extdep} {semver}\n" if extdep_listing: print( "Verify the following REQUIRED external dependencies:\n" "(Ensure their installation on all relevant systems before" " proceeding):" ) print(extdep_listing) if not confirmation_prompt("Proceed?"): return package_infos += new_pkgs prompt_for_user_vars( manager, config, configfile, args, [info for info, _, _ in package_infos] ) if not args.skiptests: # Iterate only over the requested packages here, skipping the # dependencies. We ask the manager to include dependencies below. for info, version, _ in orig_pkgs: name = info.package.qualified_name() if "test_command" not in info.metadata: zeekpkg.LOG.info( f'Skipping unit tests for "{name}": no test_command in metadata' ) continue print(f'Running unit tests for "{name}"') error_msg = "" # For testing we always process dependencies, since the tests might # well fail without them. If the user wants --nodeps and the tests # fail because of it, they'll also need to say --skiptests. error, passed, test_dir = manager.test( name, version, test_dependencies=True ) if error: error_msg = str.format("failed to run tests for {}: {}", name, error) elif not passed: clone_dir = os.path.join( os.path.join(test_dir, "clones"), info.package.name ) error_msg = str.format( '"{}" tests failed, inspect contents of' " {} for details, especially any" ' "zkg.test_command.{{stderr,stdout}}"' " files within {}", name, test_dir, clone_dir, ) if error_msg: print_error(f"error: {error_msg}") if args.force: sys.exit(1) if not confirmation_prompt( "Proceed to install anyway?", default_to_yes=False ): return installs_failed = [] for info, version, _ in reversed(package_infos): name = info.package.qualified_name() is_overwriting = False ipkg = manager.find_installed_package(name) if ipkg: is_overwriting = True modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) prev_upstream_config_files = manager.save_temporary_config_files(ipkg) worker = InstallWorker(manager, name, version) worker.start() worker.wait(f'Installing "{name}"') if worker.error: print(f'Failed installing "{name}": {worker.error}') installs_failed.append((name, version)) continue ipkg = manager.find_installed_package(name) print(f'Installed "{name}" ({ipkg.status.current_version})') if is_overwriting: for i, mf in enumerate(modifications): next_upstream_config_file = mf[1] if not os.path.isfile(next_upstream_config_file): print("\tConfig file no longer exists:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) continue prev_upstream_config_file = prev_upstream_config_files[i][1] if filecmp.cmp(prev_upstream_config_file, next_upstream_config_file): # Safe to restore user's version shutil.copy2(backup_files[i], next_upstream_config_file) continue print("\tConfig file has been overwritten with a different version:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) if manager.has_scripts(ipkg): load_error = manager.load(name) if load_error: print(f'Failed loading "{name}": {load_error}') else: print(f'Loaded "{name}"') if not args.nodeps: # Now load runtime dependencies after all dependencies and suggested # packages have been installed and loaded. for info, ver, _ in sorted(orig_pkgs, key=lambda x: x[0].package.name): _listing, saved_state = "", manager.loaded_package_states() name = info.package.qualified_name() load_error = manager.load_with_dependencies( zeekpkg.package.name_from_path(name) ) for _name, _error in load_error: if not _error: _listing += f" {_name}\n" if not _listing: dep_listing = get_changed_state(manager, saved_state, [name]) if dep_listing: print( "The following installed packages were additionally " "loaded to satisfy runtime dependencies" ) print(dep_listing) else: print( "The following installed packages could NOT be loaded " 'to satisfy runtime dependencies for "{}"'.format(name) ) print(_listing) manager.restore_loaded_package_states(saved_state) if installs_failed: print_error( "error: incomplete installation, the follow packages" " failed to be installed:" ) for n, v in installs_failed: print_error(f" {n} ({v})") sys.exit(1) def cmd_bundle(manager, args, config, configfile): packages_to_bundle = [] prefer_existing_clones = False if args.manifest: if len(args.manifest) == 1 and os.path.isfile(args.manifest[0]): config = configparser.ConfigParser(delimiters="=") config.optionxform = str if config.read(args.manifest[0]) and config.has_section("bundle"): packages = config.items("bundle") else: print_error(f'error: "{args.manifest[0]}" is not a valid manifest file') sys.exit(1) else: packages = [(name, "") for name in args.manifest] to_validate = [] new_pkgs = [] for name, version in packages: if not check_local_git_repo(name): sys.exit(1) if not version: version = active_git_branch(name) info = manager.info(name, version=version, prefer_installed=False) if info.invalid_reason: print_error( str.format( 'error: invalid package "{}": {}', name, info.invalid_reason ) ) sys.exit(1) if not version: version = info.best_version() to_validate.append((info.package.qualified_name(), version)) packages_to_bundle.append( ( info.package.qualified_name(), info.package.git_url, version, False, False, ) ) if not args.nodeps: invalid_reason, new_pkgs = manager.validate_dependencies( to_validate, ignore_installed_packages=True, ignore_suggestions=args.nosuggestions, ) if invalid_reason: print_error("error: failed to resolve dependencies:", invalid_reason) sys.exit(1) for info, version, suggested in new_pkgs: packages_to_bundle.append( ( info.package.qualified_name(), info.package.git_url, version, True, suggested, ) ) else: prefer_existing_clones = True for ipkg in manager.installed_packages(): packages_to_bundle.append( ( ipkg.package.qualified_name(), ipkg.package.git_url, ipkg.status.current_version, False, False, ) ) if not packages_to_bundle: print_error("error: no packages to put in bundle") sys.exit(1) if not args.force: package_listing = "" for name, git_url, version, is_dependency, is_suggestion in packages_to_bundle: package_listing += f" {name} ({version})" if is_suggestion: package_listing += " (suggested)" elif is_dependency: package_listing += " (dependency)" if git_url.startswith(BUILTIN_SCHEME): package_listing += " (built-in)" package_listing += "\n" print( "The following packages will be BUNDLED into {}:".format( args.bundle_filename ) ) print(package_listing) if not confirmation_prompt("Proceed?"): return git_urls = [(git_url, version) for _, git_url, version, _, _ in packages_to_bundle] error = manager.bundle( args.bundle_filename, git_urls, prefer_existing_clones=prefer_existing_clones ) if error: print_error(f"error: failed to create bundle: {error}") sys.exit(1) print(f"Bundle successfully written: {args.bundle_filename}") def cmd_unbundle(manager, args, config, configfile): prev_load_status = {} for ipkg in manager.installed_packages(): prev_load_status[ipkg.package.git_url] = ipkg.status.is_loaded if args.replace: cmd_purge(manager, args, config, configfile) error, bundle_info = manager.bundle_info(args.bundle_filename) if error: print_error(f"error: failed to unbundle {args.bundle_filename}: {error}") sys.exit(1) for git_url, version, pkg_info in bundle_info: if pkg_info.invalid_reason: name = pkg_info.package.qualified_name() print_error( "error: bundle {} contains invalid package {} ({}): {}".format( args.bundle_filename, git_url, name, pkg_info.invalid_reason ) ) sys.exit(1) if not bundle_info: print("No packages in bundle.") return if not args.force: package_listing = "" builtin_listing = "" for git_url, version, info in bundle_info: name = git_url if info.is_builtin(): builtin_listing += f" from {name} ({version}):\n" continue for pkg in manager.source_packages(): if pkg.git_url == git_url: name = pkg.qualified_name() break package_listing += f" {name} ({version})\n" print("The following packages will be INSTALLED:") print(package_listing) extdep_listing = "" for git_url, version, info in bundle_info: name = git_url for pkg in manager.source_packages(): if pkg.git_url == git_url: name = pkg.qualified_name() break extdeps = info.dependencies(field="external_depends") if extdeps is None: extdep_listing += " from {} ({}):\n \n".format( name, version ) continue if extdeps: extdep_listing += f" from {name} ({version}):\n" for extdep, semver in sorted(extdeps.items()): extdep_listing += f" {extdep} {semver}\n" if extdep_listing: print( "Verify the following REQUIRED external dependencies:\n" "(Ensure their installation on all relevant systems before" " proceeding):" ) print(extdep_listing) if not confirmation_prompt("Proceed?"): return prompt_for_user_vars( manager, config, configfile, args, [info for _, _, info in bundle_info] ) error = manager.unbundle(args.bundle_filename) if error: print_error(f"error: failed to unbundle {args.bundle_filename}: {error}") sys.exit(1) for git_url, _, _ in bundle_info: if git_url in prev_load_status: need_load = prev_load_status[git_url] else: need_load = True ipkg = manager.find_installed_package(git_url) if not ipkg: print(f'Skipped loading "{git_url}": failed to install') continue name = ipkg.package.qualified_name() if not need_load: print(f'Skipped loading "{name}"') continue load_error = manager.load(name) if load_error: print(f'Failed loading "{name}": {load_error}') else: print(f'Loaded "{name}"') print("Unbundling complete.") def cmd_remove(manager, args, config, configfile): packages_to_remove = [] def package_will_be_removed(pkg_name): for _ipkg in packages_to_remove: if _ipkg.package.name == pkg_name: return True return False for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: print_error(f'error: package "{name}" is not installed') sys.exit(1) if ipkg.is_builtin(): print_error(f'cannot remove "{name}": built-in package') sys.exit(1) packages_to_remove.append(ipkg) dependers_to_unload = set() if not args.nodeps: for ipkg in packages_to_remove: for pkg_name in manager.list_depender_pkgs(ipkg.package.name): ipkg = manager.find_installed_package(pkg_name) if ipkg and not package_will_be_removed(ipkg.package.name): if ipkg.status.is_loaded: dependers_to_unload.add(ipkg.package.name) if not args.force: print("The following packages will be REMOVED:") for ipkg in packages_to_remove: print(f" {ipkg.package.qualified_name()}") print() if dependers_to_unload: print("The following dependent packages will be UNLOADED:") for pkg_name in sorted(dependers_to_unload): ipkg = manager.find_installed_package(pkg_name) print(f" {ipkg.package.qualified_name()}") print() if not confirmation_prompt("Proceed?"): return for pkg_name in sorted(dependers_to_unload): ipkg = manager.find_installed_package(pkg_name) name = ipkg.package.qualified_name() if manager.unload(name): print(f'Unloaded "{name}"') else: # Weird that it failed, but if it's not installed and there's # nothing to unload, not worth using a non-zero exit-code to # reflect an overall failure of the package removal operation print(f'Failed unloading "{name}": no such package installed') had_failure = False for ipkg in packages_to_remove: name = ipkg.package.qualified_name() modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) if manager.remove(name): print(f'Removed "{name}"') if backup_files: print("\tCreated backups of locally modified config files:") for backup_file in backup_files: print("\t" + backup_file) else: print(f'Failed removing "{name}": no such package installed') had_failure = True if had_failure: sys.exit(1) def cmd_purge(manager, args, config, configfile): packages_to_remove = manager.installed_packages() packages_to_remove = [p for p in packages_to_remove if not p.is_builtin()] if not packages_to_remove: print("No packages to remove.") return if not args.force: package_listing = "" names_to_remove = [ipkg.package.qualified_name() for ipkg in packages_to_remove] for name in names_to_remove: package_listing += f" {name}\n" print("The following packages will be REMOVED:") print(package_listing) if not confirmation_prompt("Proceed?"): return had_failure = False for ipkg in packages_to_remove: name = ipkg.package.qualified_name() modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) if manager.remove(name): print(f'Removed "{name}"') if backup_files: print("\tCreated backups of locally modified config files:") for backup_file in backup_files: print("\t" + backup_file) else: print(f'Unknown error removing "{name}"') had_failure = True if had_failure: sys.exit(1) def outdated(manager): return [ ipkg.package.qualified_name() for ipkg in manager.installed_packages() if ipkg.status.is_outdated ] def cmd_refresh(manager, args, config, configfile): if not args.sources: args.sources = list(manager.sources.keys()) if args.fail_on_aggregate_problems and not args.aggregate: print_error( "warning: --fail-on-aggregate-problems without --aggregate" " has no effect." ) had_failure = False had_aggregation_failure = False for source in args.sources: print(f"Refresh package source: {source}") src_pkgs_before = {i.qualified_name() for i in manager.source_packages()} error = "" aggregation_issues = [] if args.aggregate: res = manager.aggregate_source(source, args.push) error = res.refresh_error aggregation_issues = res.package_issues else: error = manager.refresh_source(source, False, args.push) if error: had_failure = True print_error(f'error: failed to refresh "{source}": {error}') continue src_pkgs_after = {i.qualified_name() for i in manager.source_packages()} if src_pkgs_before == src_pkgs_after: print("\tNo membership changes") else: print("\tChanges:") diff = src_pkgs_before.symmetric_difference(src_pkgs_after) for name in diff: change = "Added" if name in src_pkgs_after else "Removed" print(f"\t\t{change} {name}") if args.aggregate: if aggregation_issues: print( "\tWARNING: Metadata aggregated, but excludes the " "following packages due to described problems:" ) for url, issue in aggregation_issues: print(f"\t\t{url}: {issue}") if args.fail_on_aggregate_problems: had_aggregation_failure = True else: print("\tMetadata aggregated") if args.push: print("\tPushed aggregated metadata") outdated_before = {i for i in outdated(manager)} print("Refresh installed packages") manager.refresh_installed_packages() outdated_after = {i for i in outdated(manager)} if outdated_before == outdated_after: print("\tNo new outdated packages") else: print("\tNew outdated packages:") diff = outdated_before.symmetric_difference(outdated_after) for name in diff: ipkg = manager.find_installed_package(name) version_change = version_change_string(manager, ipkg) print(f"\t\t{name} {version_change}") if had_failure: sys.exit(1) if had_aggregation_failure: sys.exit(2) def version_change_string(manager, installed_package): old_version = installed_package.status.current_version new_version = old_version version_change = "" if installed_package.status.tracking_method == TRACKING_METHOD_VERSION: versions = manager.package_versions(installed_package) if len(versions): new_version = versions[-1] version_change = f"({old_version} -> {new_version})" else: version_change = f"({new_version})" return version_change def cmd_upgrade(manager, args, config, configfile): if args.package: pkg_list = args.package else: pkg_list = outdated(manager) outdated_packages = [] package_listing = "" for name in pkg_list: ipkg = manager.find_installed_package(name) if not ipkg: print_error(f'error: package "{name}" is not installed') sys.exit(1) name = ipkg.package.qualified_name() if not ipkg.status.is_outdated: continue if not manager.match_source_packages(name): name = ipkg.package.git_url info = manager.info( name, version=ipkg.status.current_version, prefer_installed=False ) if info.invalid_reason: print_error( str.format('error: invalid package "{}": {}', name, info.invalid_reason) ) sys.exit(1) next_version = ipkg.status.current_version if ipkg.status.tracking_method == TRACKING_METHOD_VERSION and info.versions: next_version = info.versions[-1] outdated_packages.append((info, next_version, False)) version_change = version_change_string(manager, ipkg) package_listing += f" {name} {version_change}\n" if not outdated_packages: print("All packages already up-to-date.") return new_pkgs = [] if not args.nodeps: to_validate = [ (info.package.qualified_name(), next_version) for info, next_version, _ in outdated_packages ] invalid_reason, new_pkgs = manager.validate_dependencies( to_validate, ignore_suggestions=args.nosuggestions ) if invalid_reason: print_error("error: failed to resolve dependencies:", invalid_reason) sys.exit(1) allpkgs = outdated_packages + new_pkgs if not args.force: print("The following packages will be UPGRADED:") print(package_listing) if new_pkgs: dependency_listing = "" for info, version, suggestion in new_pkgs: name = info.package.qualified_name() dependency_listing += f" {name} ({version})" if suggestion: dependency_listing += " (suggested)" dependency_listing += "\n" print("The following dependencies will be INSTALLED:") print(dependency_listing) extdep_listing = "" for info, version, _ in allpkgs: name = info.package.qualified_name() extdeps = info.dependencies(field="external_depends") if extdeps is None: extdep_listing += " from {} ({}):\n \n".format( name, version ) continue if extdeps: extdep_listing += f" from {name} ({version}):\n" for extdep, semver in sorted(extdeps.items()): extdep_listing += f" {extdep} {semver}\n" if extdep_listing: print( "Verify the following REQUIRED external dependencies:\n" "(Ensure their installation on all relevant systems before" " proceeding):" ) print(extdep_listing) if not confirmation_prompt("Proceed?"): return prompt_for_user_vars( manager, config, configfile, args, [info for info, _, _ in allpkgs] ) if not args.skiptests: for info, version, _ in outdated_packages: name = info.package.qualified_name() # Get info object for the next version as there may have been a # test_command added during the upgrade. next_info = manager.info(name, version=version, prefer_installed=False) if next_info.invalid_reason: print_error( f'error: invalid package "{name}": {next_info.invalid_reason}' ) sys.exit(1) if "test_command" not in next_info.metadata: zeekpkg.LOG.info( 'Skipping unit tests for "%s": no test_command in metadata', name ) continue print(f'Running unit tests for "{name}"') error_msg = "" # As in cmd_install, we always process dependencies since the tests # might well fail without them. If the user wants --nodeps and the # tests fail because of it, they'll also need to say --skiptests. error, passed, test_dir = manager.test( name, version, test_dependencies=True ) if error: error_msg = str.format("failed to run tests for {}: {}", name, error) elif not passed: clone_dir = os.path.join( os.path.join(test_dir, "clones"), info.package.name ) error_msg = str.format( '"{}" tests failed, inspect contents of' " {} for details, especially any" ' "zkg.test_command.{{stderr,stdout}}"' " files within {}", name, test_dir, clone_dir, ) if error_msg: print_error(f"error: {error_msg}") if args.force: sys.exit(1) if not confirmation_prompt( "Proceed to install anyway?", default_to_yes=False ): return for info, version, _ in reversed(new_pkgs): name = info.package.qualified_name() worker = InstallWorker(manager, name, version) worker.start() worker.wait(f'Installing "{name}"') if worker.error: print(f'Failed installing "{name}": {worker.error}') continue ipkg = manager.find_installed_package(name) print(f'Installed "{name}" ({ipkg.status.current_version})') if manager.has_scripts(ipkg): load_error = manager.load(name) if load_error: print(f'Failed loading "{name}": {load_error}') else: print(f'Loaded "{name}"') had_failure = False for info, next_version, _ in outdated_packages: name = info.package.qualified_name() if not manager.match_source_packages(name): name = info.package.git_url ipkg = manager.find_installed_package(name) modifications = manager.modified_config_files(ipkg) backup_files = manager.backup_modified_files(name, modifications) prev_upstream_config_files = manager.save_temporary_config_files(ipkg) res = manager.upgrade(name) if res: print(f'Failed upgrading "{name}": {res}') had_failure = True else: ipkg = manager.find_installed_package(name) print(f'Upgraded "{name}" ({ipkg.status.current_version})') for i, mf in enumerate(modifications): next_upstream_config_file = mf[1] if not os.path.isfile(next_upstream_config_file): print("\tConfig file no longer exists:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) continue prev_upstream_config_file = prev_upstream_config_files[i][1] if filecmp.cmp(prev_upstream_config_file, next_upstream_config_file): # Safe to restore user's version shutil.copy2(backup_files[i], next_upstream_config_file) continue print("\tConfig file has been updated to a newer version:") print("\t\t" + next_upstream_config_file) print("\tPrevious, locally modified version backed up to:") print("\t\t" + backup_files[i]) if had_failure: sys.exit(1) def cmd_load(manager, args, config, configfile): had_failure = False load_error = False dep_error_listing = "" for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: had_failure = True print(f'Failed to load "{name}": no such package installed') continue if not manager.has_scripts(ipkg): print(f'The package "{name}" does not contain scripts to load.') continue name = ipkg.package.qualified_name() if args.nodeps: load_error = manager.load(name) else: saved_state = manager.loaded_package_states() dep_error_listing, load_error = "", False loaded_dep_list = manager.load_with_dependencies( zeekpkg.package.name_from_path(name) ) for _name, _error in loaded_dep_list: if _error: load_error = True dep_error_listing += f" {_name}: {_error}\n" if not load_error: dep_listing = get_changed_state(manager, saved_state, [name]) if dep_listing: print( "The following installed packages were additionally loaded to satisfy" ' runtime dependencies for "{}".'.format(name) ) print(dep_listing) if load_error: had_failure = True if not args.nodeps: if dep_error_listing: print( 'The following installed dependencies could not be loaded for "{}".'.format( name ) ) print(dep_error_listing) manager.restore_loaded_package_states(saved_state) print(f'Failed to load "{name}": {load_error}') else: print(f'Loaded "{name}"') if had_failure: sys.exit(1) def cmd_unload(manager, args, config, configfile): had_failure = False packages_to_unload = [] dependers_to_unload = set() def package_will_be_unloaded(pkg_name): for _ipkg in packages_to_unload: if _ipkg.package.name == pkg_name: return True return False for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: had_failure = True print(f'Failed to unload "{name}": no such package installed') continue if not ipkg.status.is_loaded: continue # Maybe in the future that's possible, but as of now built-in # packages are really built-in plugins and there is not a way # to unload them. if ipkg.is_builtin(): print_error(f'cannot unload "{name}": built-in package') sys.exit(1) packages_to_unload.append(ipkg) if not args.nodeps: for ipkg in packages_to_unload: for pkg_name in manager.list_depender_pkgs(ipkg.package.name): ipkg = manager.find_installed_package(pkg_name) if ipkg and not package_will_be_unloaded(ipkg.package.name): if ipkg.status.is_loaded: dependers_to_unload.add(ipkg.package.name) if packages_to_unload and not args.force: print("The following packages will be UNLOADED:") for ipkg in packages_to_unload: print(f" {ipkg.package.qualified_name()}") print() if dependers_to_unload: print("The following dependent packages will be UNLOADED:") for pkg_name in sorted(dependers_to_unload): ipkg = manager.find_installed_package(pkg_name) print(f" {ipkg.package.qualified_name()}") print() if not confirmation_prompt("Proceed?"): if had_failure: sys.exit(1) else: return for pkg_name in sorted(dependers_to_unload): packages_to_unload.append(manager.find_installed_package(pkg_name)) for ipkg in packages_to_unload: name = ipkg.package.qualified_name() if manager.unload(name): print(f'Unloaded "{name}"') else: had_failure = True print(f'Failed unloading "{name}": no such package installed') if had_failure: sys.exit(1) def cmd_pin(manager, args, config, configfile): had_failure = False for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: had_failure = True print(f'Failed to pin "{name}": no such package installed') continue if ipkg.is_builtin(): had_failure = True print_error(f'cannot pin "{name}": built-in package') continue name = ipkg.package.qualified_name() ipkg = manager.pin(name) if ipkg: print( 'Pinned "{}" at version: {} ({})'.format( name, ipkg.status.current_version, ipkg.status.current_hash ) ) else: had_failure = True print(f'Failed pinning "{name}": no such package installed') if had_failure: sys.exit(1) def cmd_unpin(manager, args, config, configfile): had_failure = False for name in args.package: ipkg = manager.find_installed_package(name) if not ipkg: had_failure = True print(f'Failed to unpin "{name}": no such package installed') continue if ipkg.is_builtin(): had_failure = True print_error(f'cannot unpin "{name}": built-in package') continue name = ipkg.package.qualified_name() ipkg = manager.unpin(name) if ipkg: print( 'Unpinned "{}" from version: {} ({})'.format( name, ipkg.status.current_version, ipkg.status.current_hash ) ) else: had_failure = True print(f'Failed unpinning "{name}": no such package installed') if had_failure: sys.exit(1) def _get_filtered_packages(manager, category): pkg_dict = dict() for ipkg in manager.installed_packages(): pkg_dict[ipkg.package.qualified_name()] = ipkg for pkg in manager.source_packages(): pkg_qn = pkg.qualified_name() if pkg_qn not in pkg_dict: pkg_dict[pkg_qn] = pkg if category == "all": filtered_pkgs = pkg_dict elif category == "installed": filtered_pkgs = { key: value for key, value in pkg_dict.items() if isinstance(value, zeekpkg.InstalledPackage) } elif category == "not_installed": filtered_pkgs = { key: value for key, value in pkg_dict.items() if not isinstance(value, zeekpkg.InstalledPackage) } elif category == "loaded": filtered_pkgs = { key: value for key, value in pkg_dict.items() if isinstance(value, zeekpkg.InstalledPackage) and value.status.is_loaded } elif category == "unloaded": filtered_pkgs = { key: value for key, value in pkg_dict.items() if isinstance(value, zeekpkg.InstalledPackage) and not value.status.is_loaded } elif category == "outdated": filtered_pkgs = { key: value for key, value in pkg_dict.items() if isinstance(value, zeekpkg.InstalledPackage) and value.status.is_outdated } else: raise NotImplementedError return filtered_pkgs def cmd_list(manager, args, config, configfile): filtered_pkgs = _get_filtered_packages(manager, args.category) for pkg_name, val in sorted(filtered_pkgs.items()): if isinstance(val, zeekpkg.InstalledPackage): pkg = val.package if val.is_builtin() and not args.include_builtin: continue out = f"{pkg_name} (installed: {val.status.current_version})" else: pkg = val out = pkg_name if not args.nodesc: desc = pkg.short_description() if desc: out += " - " + desc print(out) def cmd_search(manager, args, config, configfile): src_pkgs = manager.source_packages() matches = set() for search_text in args.search_text: if search_text[0] == "/" and search_text[-1] == "/": import re try: regex = re.compile(search_text[1:-1]) except re.error as error: print(f"invalid regex: {error}") sys.exit(1) else: for pkg in src_pkgs: if regex.search(pkg.name_with_source_directory()): matches.add(pkg) for tag in pkg.tags(): if regex.search(tag): matches.add(pkg) else: for pkg in src_pkgs: if search_text in pkg.name_with_source_directory(): matches.add(pkg) for tag in pkg.tags(): if search_text in tag: matches.add(pkg) if matches: for match in sorted(matches): out = match.qualified_name() ipkg = manager.find_installed_package(match.qualified_name()) if ipkg: out += f" (installed: {ipkg.status.current_version})" desc = match.short_description() if desc: out += " - " + desc print(out) else: print("no matches") def cmd_info(manager, args, config, configfile): if args.version and len(args.package) > 1: print_error('error: "info --version" may only be used for a single package') sys.exit(1) # Dictionary for storing package info to output as JSON pkginfo = dict() had_invalid_package = False if len(args.package) == 1: try: package_names = [] for pkg_name, info in _get_filtered_packages( manager, args.package[0] ).items(): if info.is_builtin() and not args.include_builtin: continue package_names.append(pkg_name) except NotImplementedError: package_names = args.package else: package_names = args.package for name in package_names: info = manager.info( name, version=args.version, prefer_installed=(not args.nolocal) ) if info.package: name = info.package.qualified_name() if args.json: pkginfo[name] = dict() pkginfo[name]["metadata"] = dict() else: print(f'"{name}" info:') if info.invalid_reason: if args.json: pkginfo[name]["invalid"] = info.invalid_reason else: print(f"\tinvalid package: {info.invalid_reason}\n") had_invalid_package = True continue if args.json: pkginfo[name]["url"] = info.package.git_url pkginfo[name]["versions"] = info.versions else: print(f"\turl: {info.package.git_url}") print(f"\tversions: {info.versions}") if info.status: if args.json: pkginfo[name]["install_status"] = dict() for key, value in sorted(info.status.__dict__.items()): pkginfo[name]["install_status"][key] = value else: print("\tinstall status:") for key, value in sorted(info.status.__dict__.items()): print(f"\t\t{key} = {value}") if args.json: if info.metadata_file: pkginfo[name]["metadata_file"] = info.metadata_file pkginfo[name]["metadata"][info.metadata_version] = dict() else: if info.metadata_file: print(f"\tmetadata file: {info.metadata_file}") print(f'\tmetadata (from version "{info.metadata_version}"):') if len(info.metadata) == 0: if not args.json: print("\t\t") else: if args.json: _fill_metadata_version( pkginfo[name]["metadata"][info.metadata_version], info.metadata ) else: for key, value in sorted(info.metadata.items()): value = value.replace("\n", "\n\t\t\t") print(f"\t\t{key} = {value}") # If --json and --allvers given, check for multiple versions and # add the metadata for each version to the pkginfo. if args.json and args.allvers: for vers in info.versions: # Skip the version that was already processed if vers != info.metadata_version: info2 = manager.info( name, vers, prefer_installed=(not args.nolocal) ) pkginfo[name]["metadata"][info2.metadata_version] = dict() if info2.metadata_file: pkginfo[name]["metadata_file"] = info2.metadata_file _fill_metadata_version( pkginfo[name]["metadata"][info2.metadata_version], info2.metadata, ) if not args.json: print() if args.json: print(json.dumps(pkginfo, indent=args.jsonpretty, sort_keys=True)) if had_invalid_package: sys.exit(1) def _fill_metadata_version(pkginfo_name_metadata_version, info_metadata): """Fill a dict with metadata information. This helper function is called by cmd_info to fill metadata information for a specific package version. Args: pkginfo_name_metadata_version (dict of str -> dict): Corresponds to pkginfo[name]['metadata'][info.metadata_version] in cmd_info. info_metadata (dict of str->str): Corresponds to info.metadata in cmd_info. Side effect: New dict entries are added to pkginfo_name_metadata_version. """ for key, value in info_metadata.items(): if key == "depends" or key == "suggests": pkginfo_name_metadata_version[key] = dict() deps = value.split("\n") for i in range(1, len(deps)): deplist = deps[i].split(" ") pkginfo_name_metadata_version[key][deplist[0]] = deplist[1] else: pkginfo_name_metadata_version[key] = value def cmd_config(manager, args, config, configfile): if args.config_param == "all": out = io.StringIO() config.write(out) print(out.getvalue()) out.close() elif args.config_param == "sources": for key, value in config_items(config, "sources"): print(f"{key} = {value}") elif args.config_param == "user_vars": if config.has_section("user_vars"): for key, value in config_items(config, "user_vars"): print(f"{key} = {value}") else: print(config.get("paths", args.config_param)) def cmd_autoconfig(manager, args, config, configfile): if args.user: configfile = os.path.join(home_config_dir(), "config") with open(configfile, "w", encoding=std_encoding(sys.stdout)) as f: config.write(f) print(f"Successfully wrote config file to {configfile}") return zeek_config = find_program("zeek-config") if not zeek_config: print_error('error: no "zeek-config" in PATH') sys.exit(1) cmd = subprocess.Popen( [zeek_config, "--site_dir", "--plugin_dir", "--prefix", "--zeek_dist"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True, ) script_dir = read_zeek_config_line(cmd.stdout) plugin_dir = read_zeek_config_line(cmd.stdout) bin_dir = os.path.join(read_zeek_config_line(cmd.stdout), "bin") zeek_dist = read_zeek_config_line(cmd.stdout) if configfile: config_dir = os.path.dirname(configfile) else: config_dir = default_config_dir() configfile = os.path.join(config_dir, "config") make_dir(config_dir) config_file_exists = os.path.isfile(configfile) def change_config_value(config, section, option, new_value, use_prompt): if not use_prompt: config.set(section, option, new_value) return old_value = config.get(section, option) if old_value == new_value: return prompt = f'Set "{option}" config option to: {new_value} ?' if old_value: prompt += f"\n(previous value: {old_value})" if args.force or confirmation_prompt(prompt): config.set(section, option, new_value) change_config_value(config, "paths", "script_dir", script_dir, config_file_exists) change_config_value(config, "paths", "plugin_dir", plugin_dir, config_file_exists) change_config_value(config, "paths", "bin_dir", bin_dir, config_file_exists) change_config_value(config, "paths", "zeek_dist", zeek_dist, config_file_exists) with open(configfile, "w", encoding=std_encoding(sys.stdout)) as f: config.write(f) print(f"Successfully wrote config file to {configfile}") def cmd_env(manager, args, config, configfile): zeek_config = find_program("zeek-config") zeekpath = os.environ.get("ZEEKPATH") pluginpath = os.environ.get("ZEEK_PLUGIN_PATH") if zeek_config: cmd = subprocess.Popen( [zeek_config, "--zeekpath", "--plugin_dir"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True, ) line1 = read_zeek_config_line(cmd.stdout) line2 = read_zeek_config_line(cmd.stdout) if not zeekpath: zeekpath = line1 if not pluginpath: pluginpath = line2 zeekpaths = [p for p in zeekpath.split(":")] if zeekpath else [] pluginpaths = [p for p in pluginpath.split(":")] if pluginpath else [] zeekpaths.append(manager.zeekpath()) pluginpaths.append(manager.zeek_plugin_path()) def remove_redundant_paths(paths): return list(OrderedDict.fromkeys(paths)) zeekpaths = remove_redundant_paths(zeekpaths) pluginpaths = remove_redundant_paths(pluginpaths) if os.environ.get("SHELL", "").endswith("csh"): print("setenv ZEEKPATH {}".format(":".join(zeekpaths))) print("setenv ZEEK_PLUGIN_PATH {}".format(":".join(pluginpaths))) print(f"setenv PATH {manager.bin_dir}:$PATH") else: print("export ZEEKPATH={}".format(":".join(zeekpaths))) print("export ZEEK_PLUGIN_PATH={}".format(":".join(pluginpaths))) print(f"export PATH={manager.bin_dir}:$PATH") def cmd_create(manager, args, config, configfile): tmplname = ( args.template or config.get("templates", "default", fallback=None) or ZKG_DEFAULT_TEMPLATE ) try: tmpl = Template.load(config, tmplname, args.version) except LoadError as error: msg = f"problem while loading template {tmplname}: {error}" zeekpkg.LOG.exception(msg) print_error("error: " + msg) sys.exit(1) try: package = tmpl.package() uvars = tmpl.define_user_vars() uvar_names = set(package.needed_user_vars()) # Overlay any requested features onto the package. if args.features: # If the user provided comma-separated values, split the # strings. (Argparse expects space separation.) Also # filter any duplicates. fnames = set() for feat in args.features: fnames |= {f.strip() for f in feat.split(",") if f} features = tmpl.features() for feature in features: if feature.name() in fnames: package.add_feature(feature) uvar_names |= set(feature.needed_user_vars()) fnames.remove(feature.name()) if len(fnames) > 0: # Alert if the user requested an unknown feature. knowns = ", ".join([f'"{f.name()}"' for f in features]) unknowns = ", ".join([f'"{name}"' for name in fnames]) print_error( "error: the following features are unknown: {}." ' Template "{}" offers {}.'.format( unknowns, tmpl.name(), knowns or "no features" ) ) sys.exit(1) # Remove user vars we don't actually require from consideration uvars = [uvar for uvar in uvars if uvar.name() in uvar_names] # Resolve the variables via user input, args, etc for uvar in uvars: try: uvar.resolve(tmpl.name(), config, args.user_var, args.force) except ValueError: print_error( str.format( 'error: could not determine value of user variable "{}",' " provide via environment or --user-var", uvar.name(), ) ) sys.exit(1) # Apply them to the template. After this, any parameter can be # retrieved from the template via tmpl.lookup_param(). tmpl._set_user_vars(uvars) # Verify that resulting template parameters are formatted correctly try: package.do_validate(tmpl) except zeekpkg.template.InputError as error: print_error("error: template input invalid, " + str(error)) sys.exit(1) # And finally, instantiate the package. try: if os.path.isdir(args.packagedir): if not args.force: print(f"Package directory {args.packagedir} already exists.") if not confirmation_prompt("Delete?"): sys.exit(1) try: delete_path(args.packagedir) zeekpkg.LOG.info( "Removed existing package directory %s", args.packagedir ) except OSError as err: print_error( "error: could not remove package directory {}: {}".format( args.packagedir, err ) ) sys.exit(1) package.do_instantiate(tmpl, args.packagedir, args.force) except zeekpkg.template.OutputError as error: print_error("error: template instantiation failed, " + str(error)) sys.exit(1) except Exception as error: msg = f"problem during template instantiation: {error}" zeekpkg.LOG.exception(msg) print_error("error: " + msg) sys.exit(1) def cmd_template_info(manager, args, config, configfile): tmplname = ( args.template or config.get("templates", "default", fallback=None) or ZKG_DEFAULT_TEMPLATE ) try: tmpl = Template.load(config, tmplname, args.version) except LoadError as error: msg = f"problem while loading template {tmplname}: {error}" zeekpkg.LOG.exception(msg) print_error("error: " + msg) sys.exit(1) tmplinfo = tmpl.info() if args.json: print(json.dumps(tmplinfo, indent=args.jsonpretty, sort_keys=True)) else: print("API version: " + tmplinfo["api_version"]) print("features: " + ", ".join(tmplinfo["features"])) print("origin: " + tmplinfo["origin"]) print("provides package: " + str(tmplinfo["provides_package"]).lower()) print("user vars:") for uvar_name, uvar_info in tmplinfo["user_vars"].items(): print( "\t{}: {}, {}, used by {}".format( uvar_name, uvar_info["description"], uvar_info["default"] or "no default", ", ".join(uvar_info["used_by"]) or "not used", ) ) print("versions: " + ", ".join(tmplinfo["versions"])) class BundleHelpFormatter(argparse.ArgumentDefaultsHelpFormatter): # Workaround for underlying argparse bug: https://bugs.python.org/issue9338 def _format_args(self, action, default_metavar): rval = super()._format_args(action, default_metavar) if action.nargs == argparse.ZERO_OR_MORE: rval += " --" elif action.nargs == argparse.ONE_OR_MORE: rval += " --" return rval def top_level_parser(): top_parser = argparse.ArgumentParser( formatter_class=argparse.RawDescriptionHelpFormatter, description="A command-line package manager for Zeek.", epilog="Environment Variables:\n\n" " ``ZKG_CONFIG_FILE``:\t" "Same as ``--configfile`` option, but has less precedence.\n" " ``ZKG_DEFAULT_SOURCE``:\t" "The default package source to use (normally {}).\n" " ``ZKG_DEFAULT_TEMPLATE``:\t" "The default package template to use (normally {}).\n".format( ZKG_DEFAULT_SOURCE, ZKG_DEFAULT_TEMPLATE ), ) top_parser.add_argument( "--version", action="version", version="%(prog)s " + zeekpkg.__version__ ) group = top_parser.add_mutually_exclusive_group() group.add_argument( "--configfile", metavar="FILE", help="Path to Zeek Package Manager config file. Precludes --user.", ) group.add_argument( "--user", action="store_true", help="Store all state in user's home directory. Precludes --configfile.", ) top_parser.add_argument( "--verbose", "-v", action="count", default=0, help="Increase program output for debugging." " Use multiple times for more output (e.g. -vv).", ) top_parser.add_argument( "--extra-source", action="append", metavar="NAME=URL", help="Add an extra source.", ) return top_parser def argparser(): pkg_name_help = ( "The name(s) of package(s) to operate on. The package" " may be named in several ways. If the package is part" " of a package source, it may be referred to by the" " base name of the package (last component of git URL)" " or its path within the package source." " If two packages in different package sources" " have conflicting paths, then the package source" " name may be prepended to the package path to resolve" " the ambiguity. A full git URL may also be used to refer" " to a package that does not belong to a source. E.g. for" ' a package source called "zeek" that has a package named' ' "foo" located in "alice/zkg.index", the following' ' names work: "foo", "alice/foo", "zeek/alice/foo".' ) def add_uservar_args(parser, force_help=None): parser.add_argument( "--force", action="store_true", help=force_help or "Don't prompt for confirmation or user variables.", ) parser.add_argument( "--user-var", action="append", metavar="NAME=VAL", type=UserVar.parse_arg, help="A user variable assignment. This avoids prompting" " for input and lets you provide a value when using --force." " Use repeatedly as needed for multiple values.", ) def add_json_args(parser, help_text): parser.add_argument("--json", action="store_true", help=help_text) parser.add_argument( "--jsonpretty", type=int, default=None, metavar="SPACES", help="Optional number of spaces to indent for pretty-printed" " JSON output.", ) top_parser = top_level_parser() command_parser = top_parser.add_subparsers( title="commands", dest="command", help="See `%(prog)s -h` for per-command usage info.", ) command_parser.required = True # test sub_parser = command_parser.add_parser( "test", help="Runs unit tests for Zeek packages.", description="Runs the unit tests for the specified Zeek packages." ' In most cases, the "zeek" and "zeek-config" programs will' " need to be in PATH before running this command.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_test) sub_parser.add_argument("package", nargs="+", help=pkg_name_help) sub_parser.add_argument( "--version", default=None, help="The version of the package to test. Only one package may be" " specified at a time when using this flag. A version tag, branch" " name, or commit hash may be specified here." " If the package name refers to a local git repo with a working tree," " then its currently active branch is used." " The default for other cases is to use" " the latest version tag, or if a package has none," ' the default branch, like "main" or "master".', ) # install sub_parser = command_parser.add_parser( "install", help="Installs Zeek packages.", description="Installs packages from a configured package source or" " directly from a git URL. After installing, the package" ' is marked as being "loaded" (see the ``load`` command).', formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_install) sub_parser.add_argument("package", nargs="+", help=pkg_name_help) sub_parser.add_argument( "--skiptests", action="store_true", help="Skip running unit tests for packages before installation.", ) sub_parser.add_argument( "--nodeps", action="store_true", help="Skip all dependency resolution/checks. Note that using this" " option risks putting your installed package collection into a" " broken or unusable state.", ) sub_parser.add_argument( "--nosuggestions", action="store_true", help="Skip automatically installing suggested packages.", ) sub_parser.add_argument( "--version", default=None, help="The version of the package to install. Only one package may be" " specified at a time when using this flag. A version tag, branch" " name, or commit hash may be specified here." " If the package name refers to a local git repo with a working tree," " then its currently active branch is used." " The default for other cases is to use" " the latest version tag, or if a package has none," ' the default branch, like "main" or "master".', ) add_uservar_args(sub_parser) # bundle sub_parser = command_parser.add_parser( "bundle", help="Creates a bundle file containing a collection of Zeek packages.", description="This command creates a bundle file containing a collection" " of Zeek packages. If ``--manifest`` is used, the user" " supplies the list of packages to put in the bundle, else" " all currently installed packages are put in the bundle." " A bundle file can be unpacked on any target system," " resulting in a repeatable/specific set of packages" " being installed on that target system (see the" " ``unbundle`` command). This command may be useful for" " those that want to manage packages on a system that" " otherwise has limited network connectivity. E.g. one can" " use a system with an internet connection to create a" " bundle, transport that bundle to the target machine" " using whatever means are appropriate, and finally" " unbundle/install it on the target machine.", formatter_class=BundleHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_bundle) sub_parser.add_argument( "bundle_filename", metavar="filename.bundle", help="The path of the bundle file to create. It will be overwritten" " if it already exists. Note that if --manifest is used before" " this filename is specified, you should use a double-dash, --," " to first terminate that argument list.", ) sub_parser.add_argument( "--force", action="store_true", help="Skip the confirmation prompt." ) sub_parser.add_argument( "--nodeps", action="store_true", help="Skip all dependency resolution/checks. Note that using this" " option risks creating a bundle of packages that is in a" " broken or unusable state.", ) sub_parser.add_argument( "--nosuggestions", action="store_true", help="Skip automatically bundling suggested packages.", ) sub_parser.add_argument( "--manifest", nargs="+", help="This may either be a file name or a list of packages to include" " in the bundle. If a file name is supplied, it should be in INI" " format with a single ``[bundle]`` section. The keys in that section" " correspond to package names and their values correspond to git" " version tags, branch names, or commit hashes. The values may be" " left blank to indicate that the latest available version should be" " used.", ) # unbundle sub_parser = command_parser.add_parser( "unbundle", help="Unpacks Zeek packages from a bundle file and installs them.", description="This command unpacks a bundle file formerly created by the" " ``bundle`` command and installs all the packages" " contained within.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_unbundle) sub_parser.add_argument( "bundle_filename", metavar="filename.bundle", help="The path of the bundle file to install.", ) sub_parser.add_argument( "--replace", action="store_true", help="Using this flag first removes all installed packages before then" " installing the packages from the bundle.", ) add_uservar_args(sub_parser) # remove sub_parser = command_parser.add_parser( "remove", aliases=["uninstall"], help="Uninstall a package.", description="Unloads (see the ``unload`` command) and uninstalls a" " previously installed package.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_remove) sub_parser.add_argument("package", nargs="+", help=pkg_name_help) sub_parser.add_argument( "--force", action="store_true", help="Skip the confirmation prompt." ) sub_parser.add_argument( "--nodeps", action="store_true", help="Skip all dependency resolution/checks. Note that using this" " option risks putting your installed package collection into a" " broken or unusable state.", ) # purge sub_parser = command_parser.add_parser( "purge", help="Uninstall all packages.", description="Unloads (see the ``unload`` command) and uninstalls all" " previously installed packages.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_purge) sub_parser.add_argument( "--force", action="store_true", help="Skip the confirmation prompt." ) # refresh sub_parser = command_parser.add_parser( "refresh", help="Retrieve updated package metadata.", description="Retrieve latest package metadata from sources and checks" " whether any installed packages have available upgrades." " Note that this does not actually upgrade any packages (see the" " ``upgrade`` command for that).", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_refresh) sub_parser.add_argument( "--aggregate", action="store_true", help="Crawls the urls listed in package source zkg.index files and" " aggregates the metadata found in their zkg.meta (or legacy" " bro-pkg.meta) files. The aggregated metadata is stored in the local" " clone of the package source that zkg uses internally for locating" " package metadata." " For each package, the metadata is taken from the highest available" ' git version tag or the default branch, like "main" or "master", if no version tags exist', ) sub_parser.add_argument( "--fail-on-aggregate-problems", action="store_true", help="When using --aggregate, exit with error when any packages trigger" " metadata problems. Normally such problems only cause a warning.", ) sub_parser.add_argument( "--push", action="store_true", help="Push all local changes to package sources to upstream repos", ) sub_parser.add_argument( "--sources", nargs="+", help="A list of package source names to operate on. If this argument" " is not used, then the command will operate on all configured" " sources.", ) # upgrade sub_parser = command_parser.add_parser( "upgrade", help="Upgrade installed packages to latest versions.", description="Uprades the specified package(s) to latest available" " version. If no specific packages are specified, then all installed" " packages that are outdated and not pinned are upgraded. For packages" " that are installed with ``--version`` using a git branch name, the" " package is updated to the latest commit on that branch, else the" " package is updated to the highest available git version tag.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_upgrade) sub_parser.add_argument("package", nargs="*", default=[], help=pkg_name_help) sub_parser.add_argument( "--skiptests", action="store_true", help="Skip running unit tests for packages before installation.", ) sub_parser.add_argument( "--nodeps", action="store_true", help="Skip all dependency resolution/checks. Note that using this" " option risks putting your installed package collection into a" " broken or unusable state.", ) sub_parser.add_argument( "--nosuggestions", action="store_true", help="Skip automatically installing suggested packages.", ) add_uservar_args(sub_parser) # load sub_parser = command_parser.add_parser( "load", help="Register packages to be be auto-loaded by Zeek.", description="The Zeek Package Manager keeps track of all packages that" ' are marked as "loaded" and maintains a single Zeek script that, when' " loaded by Zeek (e.g. via ``@load packages``), will load the scripts" ' from all "loaded" packages at once.' ' This command adds a set of packages to the "loaded packages" list.', formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_load) sub_parser.add_argument( "package", nargs="+", default=[], help="Name(s) of package(s) to load." ) sub_parser.add_argument( "--nodeps", action="store_true", help="Skip all dependency resolution/checks. Note that using this" " option risks putting your installed package collection into a" " broken or unusable state.", ) # unload sub_parser = command_parser.add_parser( "unload", help="Unregister packages to be be auto-loaded by Zeek.", description="The Zeek Package Manager keeps track of all packages that" ' are marked as "loaded" and maintains a single Zeek script that, when' ' loaded by Zeek, will load the scripts from all "loaded" packages at' ' once. This command removes a set of packages from the "loaded' ' packages" list.', formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_unload) sub_parser.add_argument("package", nargs="+", default=[], help=pkg_name_help) sub_parser.add_argument( "--force", action="store_true", help="Skip the confirmation prompt." ) sub_parser.add_argument( "--nodeps", action="store_true", help="Skip all dependency resolution/checks. Note that using this" " option risks putting your installed package collection into a" " broken or unusable state.", ) # pin sub_parser = command_parser.add_parser( "pin", help="Prevent packages from being automatically upgraded.", description="Pinned packages are ignored by the ``upgrade`` command.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_pin) sub_parser.add_argument("package", nargs="+", default=[], help=pkg_name_help) # unpin sub_parser = command_parser.add_parser( "unpin", help="Allows packages to be automatically upgraded.", description="Packages that are not pinned are automatically upgraded" " by the ``upgrade`` command", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_unpin) sub_parser.add_argument("package", nargs="+", default=[], help=pkg_name_help) # list sub_parser = command_parser.add_parser( "list", help="Lists packages.", description="Outputs a list of packages that match a given category.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_list) sub_parser.add_argument( "category", nargs="?", default="installed", choices=["all", "installed", "not_installed", "loaded", "unloaded", "outdated"], help="Package category used to filter listing.", ) sub_parser.add_argument( "--nodesc", action="store_true", help="Do not display description text, just the package name(s).", ) sub_parser.add_argument( "--include-builtin", action="store_true", help="Also output packages that Zeek has built-in. By default" " these are not shown.", ) # search sub_parser = command_parser.add_parser( "search", help="Search packages for matching names.", description="Perform a substring search on package names and metadata" " tags. Surround search text with slashes to indicate it is a regular" " expression (e.g. ``/text/``).", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_search) sub_parser.add_argument( "search_text", nargs="+", default=[], help="The text(s) or pattern(s) to look for.", ) # info sub_parser = command_parser.add_parser( "info", help="Display package information.", description="Shows detailed information/metadata for given packages." " If the package is currently installed, additional information about" " the status of it is displayed. E.g. the installed version or whether" ' it is currently marked as "pinned" or "loaded."', formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_info) sub_parser.add_argument( "package", nargs="+", default=[], help=pkg_name_help + " If a single name is given and matches one of the same categories" ' as the "list" command, then it is automatically expanded to be the' " names of all packages which match the given category.", ) sub_parser.add_argument( "--version", default=None, help="The version of the package metadata to inspect. A version tag," " branch name, or commit hash and only one package at a time may be" " given when using this flag. If unspecified, the behavior depends" " on whether the package is currently installed. If installed," " the metadata will be pulled from the installed version. If not" " installed, the latest version tag is used, or if a package has no" ' version tags, the default branch, like "main" or "master", is used.', ) sub_parser.add_argument( "--nolocal", action="store_true", help="Do not read information from locally installed packages." " Instead read info from remote GitHub.", ) sub_parser.add_argument( "--include-builtin", action="store_true", help="Also output packages that Zeek has built-in. By default" " these are not shown.", ) add_json_args(sub_parser, "Output package information as JSON.") sub_parser.add_argument( "--allvers", action="store_true", help="When outputting package information as JSON, show metadata for" " all versions. This option can be slow since remote repositories" " may be cloned multiple times. Also, installed packages will show" " metadata only for the installed version unless the --nolocal " " option is given.", ) # config sub_parser = command_parser.add_parser( "config", help="Show Zeek Package Manager configuration info.", description="The default output of this command is a valid package" " manager config file that corresponds to the one currently being used," " but also with any defaulted field values filled in. This command" " also allows for only the value of a specific field to be output if" " the name of that field is given as an argument to the command.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_config) sub_parser.add_argument( "config_param", nargs="?", default="all", choices=[ "all", "sources", "user_vars", "state_dir", "script_dir", "plugin_dir", "bin_dir", "zeek_dist", ], help="Name of a specific config file field to output.", ) # autoconfig sub_parser = command_parser.add_parser( "autoconfig", help="Generate a Zeek Package Manager configuration file.", description="The output of this command is a valid package manager" " config file that is generated by using the ``zeek-config`` script" " that is installed along with Zeek. It is the suggested configuration" " to use for most Zeek installations. For this command to work, the" " ``zeek-config`` script must be in ``PATH``," " unless the --user option is given, in which case this creates" " a config that does not touch the Zeek installation.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_autoconfig) sub_parser.add_argument( "--force", action="store_true", help="Skip any confirmation prompt." ) # env sub_parser = command_parser.add_parser( "env", help="Show the value of environment variables that need to be set for" " Zeek to be able to use installed packages.", description="This command returns shell commands that, when executed," " will correctly set ``ZEEKPATH`` and ``ZEEK_PLUGIN_PATH`` to use" " scripts and plugins from packages installed by the package manager." " For this command to function properly, either have the ``zeek-config``" " script (installed by zeek) in ``PATH``, or have the ``ZEEKPATH`` and" " ``ZEEK_PLUGIN_PATH`` environment variables already set so this command" " can append package-specific paths to them.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.set_defaults(run_cmd=cmd_env) # create sub_parser = command_parser.add_parser( "create", help="Create a new Zeek package.", description="This command creates a new Zeek package in the directory" " provided via --packagedir. If this directory exists, zkg will not" " modify it unless you provide --force.", formatter_class=argparse.ArgumentDefaultsHelpFormatter, ) sub_parser.add_argument( "--packagedir", metavar="DIR", required=True, help="Output directory into which to produce the new package. Required.", ) sub_parser.add_argument( "--version", help="The template version to use. A version tag, branch name, or" " commit hash may be specified here. If --template refers to a local" " git repo with a working tree, then zkg uses it as-is and the version" " is ignored. The default for other cases is to use the latest" " version tag, or if a template has none, the default branch, like" ' "main" or "master".', ) sub_parser.add_argument( "--features", nargs="+", metavar="FEATURE", help="Additional features to include in your package. Use the ``template" " info`` command for information about available features.", ) sub_parser.add_argument( "--template", metavar="URL", help="By default, zkg uses its own package template. This makes it" " select an alternative.", ) sub_parser.set_defaults(run_cmd=cmd_create) add_uservar_args(sub_parser) # Template management commands sub_parser = command_parser.add_parser("template", help="Manage package templates.") template_command_parser = sub_parser.add_subparsers( title="template commands", dest="command", help="See %(prog)s -h for per-command usage info.", ) template_command_parser.required = True # template info sub_parser = template_command_parser.add_parser( "info", help="Shows information about a package template.", description="This command shows versions and supported features for" " a given package.", ) add_json_args(sub_parser, "Output template information as JSON.") sub_parser.add_argument( "--version", help="The template version to report on. A version tag, branch name," " or commit hash may be specified here. If the selected template" " refers to a local git repo, the version is ignored. The default" " for other cases is to use the latest version tag, or if a template" ' has none, the default branch, like "main" or "master".', ) sub_parser.add_argument( "template", metavar="URL", nargs="?", help="URL of a package template repository, or local path to one." " When not provided, the configured default template is used.", ) sub_parser.set_defaults(run_cmd=cmd_template_info) if "argcomplete" in sys.modules: argcomplete.autocomplete(top_parser) return top_parser def main(): args = argparser().parse_args() formatter = logging.Formatter( "%(asctime)s %(levelname)-8s %(message)s", "%Y-%m-%d %H:%M:%S" ) handler = logging.StreamHandler() handler.setFormatter(formatter) if args.verbose == 1: zeekpkg.LOG.setLevel(logging.INFO) elif args.verbose >= 2: zeekpkg.LOG.setLevel(logging.DEBUG) zeekpkg.LOG.addHandler(handler) configfile = args.configfile if not configfile: configfile = find_configfile(args) config = create_config(args, configfile) manager = create_manager(args, config) args.run_cmd(manager, args, config, configfile) if __name__ == "__main__": main() sys.exit(0) package-manager-3.0.1/.gitignore0000644000175000017500000000006414565163601014623 0ustar rharha*.pyc *.egg-info *.swp .state doc/_build build dist package-manager-3.0.1/zkg.py0000777000175000017500000000000014565163601014513 2zkgustar rharhapackage-manager-3.0.1/requirements.txt0000644000175000017500000000022714565163601016120 0ustar rharha# Requirements for general zkg usage GitPython semantic_version btest # Requirements for development (e.g. building docs) Sphinx>=3.0 sphinx_rtd_theme package-manager-3.0.1/README.rst0000777000175000017500000000000014565163601017464 2doc/overview.rstustar rharhapackage-manager-3.0.1/setup.py0000644000175000017500000000224714565163601014352 0ustar rharhaimport pathlib from setuptools import setup install_requires = ["gitpython", "semantic_version", "btest"] def version(): return pathlib.Path("VERSION").read_text().replace("-", ".dev", 1).strip() def long_description(): return pathlib.Path("README").read_text() setup( name="zkg", version=version(), description="The Zeek Package Manager", long_description=long_description(), license="University of Illinois/NCSA Open Source License", keywords="zeek zeekctl zeekcontrol package manager scripts plugins security", maintainer="The Zeek Project", maintainer_email="info@zeek.org", url="https://github.com/zeek/package-manager", scripts=["zkg"], packages=["zeekpkg"], install_requires=install_requires, classifiers=[ "Development Status :: 5 - Production/Stable", "Environment :: Console", "License :: OSI Approved :: University of Illinois/NCSA Open Source License", "Operating System :: POSIX :: Linux", "Operating System :: MacOS :: MacOS X", "Programming Language :: Python :: 3", "Topic :: System :: Networking :: Monitoring", "Topic :: Utilities", ], ) package-manager-3.0.1/CMakeLists.txt0000644000175000017500000000231214565163601015371 0ustar rharhacmake_minimum_required(VERSION 3.5 FATAL_ERROR) project(zkg) set(ZEEK_BIN_DIR ${CMAKE_INSTALL_PREFIX}/bin) set(orig_file ${CMAKE_CURRENT_SOURCE_DIR}/zkg) set(configed_file ${CMAKE_CURRENT_BINARY_DIR}/zkg) configure_file(${orig_file} ${configed_file} @ONLY) # Install zkg install(DIRECTORY DESTINATION bin) install(PROGRAMS ${configed_file} DESTINATION bin) if ( NOT PY_MOD_INSTALL_DIR ) # This is not a Zeek-bundled install. Default to "home"-style install. set(PY_MOD_INSTALL_DIR lib/python) endif () # Install the Python module tree install(DIRECTORY DESTINATION ${PY_MOD_INSTALL_DIR}) install(DIRECTORY zeekpkg DESTINATION ${PY_MOD_INSTALL_DIR}) if ( NOT ZEEK_MAN_INSTALL_PATH ) set(ZEEK_MAN_INSTALL_PATH ${CMAKE_INSTALL_PREFIX}/share/man) endif () install(FILES doc/man/zkg.1 DESTINATION ${ZEEK_MAN_INSTALL_PATH}/man1) message( "\n=====================| ZKG Build Summary |====================" "\n" "\nInstall prefix: ${CMAKE_INSTALL_PREFIX}" "\nPython module path: ${PY_MOD_INSTALL_DIR}" "\nConfig path: ${ZEEK_ZKG_CONFIG_DIR}" "\nState path: ${ZEEK_ZKG_STATE_DIR}" "\n" "\n================================================================\n" ) package-manager-3.0.1/testing/0000755000175000017500000000000014565163601014310 5ustar rharhapackage-manager-3.0.1/testing/sources/0000755000175000017500000000000014565163601015773 5ustar rharhapackage-manager-3.0.1/testing/sources/one/0000755000175000017500000000000014565163601016554 5ustar rharhapackage-manager-3.0.1/testing/sources/one/bob/0000755000175000017500000000000014565163601017316 5ustar rharhapackage-manager-3.0.1/testing/sources/one/bob/zkg.index0000644000175000017500000000001514565163601021136 0ustar rharhacorge grault package-manager-3.0.1/testing/sources/one/alice/0000755000175000017500000000000014565163601017631 5ustar rharhapackage-manager-3.0.1/testing/sources/one/alice/zkg.index0000644000175000017500000000004214565163601021451 0ustar rharhafoo bar baz qux i-have-no-scripts package-manager-3.0.1/testing/sources/two/0000755000175000017500000000000014565163601016604 5ustar rharhapackage-manager-3.0.1/testing/sources/two/eve/0000755000175000017500000000000014565163601017363 5ustar rharhapackage-manager-3.0.1/testing/sources/two/eve/zkg.index0000644000175000017500000000000514565163601021202 0ustar rharhaquux package-manager-3.0.1/testing/templates/0000755000175000017500000000000014565163601016306 5ustar rharhapackage-manager-3.0.1/testing/templates/foo/0000755000175000017500000000000014565163601017071 5ustar rharhapackage-manager-3.0.1/testing/templates/foo/readme/0000755000175000017500000000000014565163601020326 5ustar rharhapackage-manager-3.0.1/testing/templates/foo/readme/README.md0000644000175000017500000000004514565163601021604 0ustar rharhaThis is the @NAME@ package. @README@ package-manager-3.0.1/testing/templates/foo/readme/README0000777000175000017500000000000014565163601022462 2README.mdustar rharhapackage-manager-3.0.1/testing/templates/foo/__init__.py0000644000175000017500000000243414565163601021205 0ustar rharhaimport zeekpkg.template import zeekpkg.uservar TEMPLATE_API_VERSION = "1.0.0" class Package(zeekpkg.template.Package): def contentdir(self): return "package" def needed_user_vars(self): return ["name"] def validate(self, tmpl): if not tmpl.lookup_param("name"): raise zeekpkg.template.InputError("package requires a name") class Readme(zeekpkg.template.Feature): def contentdir(self): return "readme" def needed_user_vars(self): return ["readme"] class Template(zeekpkg.template.Template): def define_user_vars(self): return [ zeekpkg.uservar.UserVar( "name", desc='the name of the package, e.g. "FooBar"' ), zeekpkg.uservar.UserVar( "readme", desc="Content of the README file", val="This is a README." ), ] def apply_user_vars(self, uvars): for uvar in uvars: if uvar.name() == "name": self.define_param("name", uvar.val()) self.define_param("module", uvar.val().upper()) if uvar.name() == "readme": self.define_param("readme", uvar.val()) def package(self): return Package() def features(self): return [Readme()] package-manager-3.0.1/testing/templates/foo/package/0000755000175000017500000000000014565163601020464 5ustar rharhapackage-manager-3.0.1/testing/templates/foo/package/scripts/0000755000175000017500000000000014565163601022153 5ustar rharhapackage-manager-3.0.1/testing/templates/foo/package/scripts/main.zeek0000644000175000017500000000010114565163601023747 0ustar rharhamodule @MODULE@; event zeek_init() { print "Hello world!"; } package-manager-3.0.1/testing/templates/foo/package/scripts/__load__.zeek0000644000175000017500000000001514565163601024542 0ustar rharha@load ./main package-manager-3.0.1/testing/templates/foo/package/zkg.meta0000644000175000017500000000030114565163601022121 0ustar rharha[package] script_dir = scripts summary = TODO: A summary of @NAME@ in one line description = TODO: A more detailed description of @NAME@. It can span multiple lines, with this indentation. package-manager-3.0.1/testing/packages/0000755000175000017500000000000014565163601016066 5ustar rharhapackage-manager-3.0.1/testing/packages/baz/0000755000175000017500000000000014565163601016642 5ustar rharhapackage-manager-3.0.1/testing/packages/baz/zkg.meta0000644000175000017500000000001214565163601020276 0ustar rharha[package] package-manager-3.0.1/testing/packages/baz/__load__.zeek0000644000175000017500000000002414565163601021231 0ustar rharhaprint "baz loaded"; package-manager-3.0.1/testing/packages/foo/0000755000175000017500000000000014565163601016651 5ustar rharhapackage-manager-3.0.1/testing/packages/foo/zkg.meta0000644000175000017500000000001214565163601020305 0ustar rharha[package] package-manager-3.0.1/testing/packages/foo/__load__.zeek0000644000175000017500000000002414565163601021240 0ustar rharhaprint "foo loaded"; package-manager-3.0.1/testing/packages/rot13/0000755000175000017500000000000014565163601017036 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/src/0000755000175000017500000000000014565163601017625 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/src/Plugin.cc0000644000175000017500000000054614565163601021377 0ustar rharha #include "Plugin.h" namespace plugin { namespace Demo_Rot13 { Plugin plugin; } } using namespace plugin::Demo_Rot13; zeek::plugin::Configuration Plugin::Configure() { zeek::plugin::Configuration config; config.name = "Demo::Rot13"; config.description = ""; config.version.major = 0; config.version.minor = 1; return config; } package-manager-3.0.1/testing/packages/rot13/src/rot13.bif0000644000175000017500000000077214565163601021265 0ustar rharha%%{ #include #include #include "zeek/util.h" #include "zeek/ZeekString.h" #include "zeek/Val.h" %%} module Demo; function rot13%(s: string%) : string %{ char* rot13 = util::copy_string(s->CheckString()); for ( char* p = rot13; *p; p++ ) { char b = islower(*p) ? 'a' : 'A'; *p = (*p - b + 13) % 26 + b; } auto zs = new zeek::String(1, reinterpret_cast(rot13), strlen(rot13)); return make_intrusive(zs); %} package-manager-3.0.1/testing/packages/rot13/src/Plugin.h0000644000175000017500000000041014565163601021227 0ustar rharha #pragma once #include namespace plugin { namespace Demo_Rot13 { class Plugin : public zeek::plugin::Plugin { protected: // Overridden from plugin::Plugin. virtual zeek::plugin::Configuration Configure(); }; extern Plugin plugin; } } package-manager-3.0.1/testing/packages/rot13/configure.plugin0000644000175000017500000000057414565163601022245 0ustar rharha#!/bin/sh # # Hooks to add custom options to the configure script. # plugin_usage() { : # Do nothing # cat < Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: (1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. (2) Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. (3) Neither the name of , nor the names of contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. package-manager-3.0.1/testing/packages/rot13/scripts/0000755000175000017500000000000014565163601020525 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/scripts/init.zeek0000644000175000017500000000000014565163601022336 0ustar rharhapackage-manager-3.0.1/testing/packages/rot13/scripts/Demo/0000755000175000017500000000000014565163601021411 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/scripts/Demo/Rot13/0000755000175000017500000000000014565163601022321 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/scripts/Demo/Rot13/__load__.zeek0000644000175000017500000000027214565163601024715 0ustar rharha# # This is loaded when a user activates the plugin. Include scripts here that should be # loaded automatically at that point. # event zeek_init() { print "rot13 script is loaded"; } package-manager-3.0.1/testing/packages/rot13/scripts/__preload__.zeek0000644000175000017500000000055014565163601023627 0ustar rharha# # This is loaded unconditionally at Zeek startup before any of the BiFs that the # plugin defines become available. # # This is primarily for defining types that BiFs already depend on. If you need # to do any other unconditional initialization (usually that's just for other BiF # elemets), that should go into __load__.zeek instead. # @load ./types.zeek package-manager-3.0.1/testing/packages/rot13/scripts/__load__.zeek0000644000175000017500000000054214565163601023121 0ustar rharha# # This is loaded unconditionally at Zeek startup. Include scripts here that should # always be loaded. # # Normally, that will be only code that initializes built-in elements. Load # your standard scripts in # scripts///__load__.zeek instead. # @load ./init.zeek event zeek_init() { print "rot13 plugin is loaded"; } package-manager-3.0.1/testing/packages/rot13/scripts/types.zeek0000644000175000017500000000000014565163601022537 0ustar rharhapackage-manager-3.0.1/testing/packages/rot13/CMakeLists.txt0000644000175000017500000000036114565163601021576 0ustar rharha cmake_minimum_required(VERSION 3.15) project(Plugin) include(ZeekPlugin) zeek_plugin_begin(Demo Rot13) zeek_plugin_cc(src/Plugin.cc) zeek_plugin_bif(src/rot13.bif) zeek_plugin_dist_files(README CHANGES COPYING VERSION) zeek_plugin_end() package-manager-3.0.1/testing/packages/rot13/testing/0000755000175000017500000000000014565163601020513 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/testing/btest.cfg0000644000175000017500000000000014565163601022303 0ustar rharhapackage-manager-3.0.1/testing/packages/rot13/testing/Baseline/0000755000175000017500000000000014565163601022235 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/testing/Baseline/tests.rot13/0000755000175000017500000000000014565163601024346 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/testing/Baseline/tests.rot13/output0000644000175000017500000000003514565163601025627 0ustar rharhaUryyb rot13 plugin is loaded package-manager-3.0.1/testing/packages/rot13/testing/Baseline/tests.main/0000755000175000017500000000000014565163601024322 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/testing/Baseline/tests.main/output0000644000175000017500000000005614565163601025606 0ustar rharharot13 plugin is loaded rot13 script is loaded package-manager-3.0.1/testing/packages/rot13/testing/tests/0000755000175000017500000000000014565163601021655 5ustar rharhapackage-manager-3.0.1/testing/packages/rot13/testing/tests/rot130000644000175000017500000000014014565163601022543 0ustar rharha# @TEST-EXEC: zeek %INPUT > output # @TEST-EXEC: btest-diff output print Demo::rot13("Hello"); package-manager-3.0.1/testing/packages/rot13/testing/tests/main0000644000175000017500000000014114565163601022520 0ustar rharha# @TEST-EXEC: zeek rot13 > output # @TEST-EXEC: TEST_DIFF_CANONIFIER="sort -s" btest-diff output package-manager-3.0.1/testing/packages/rot13/VERSION0000644000175000017500000000000414565163601020100 0ustar rharha0.1 package-manager-3.0.1/testing/packages/rot13/Makefile0000644000175000017500000000130014565163601020470 0ustar rharha# # Convenience Makefile providing a few common top-level targets. # cmake_build_dir=build arch=`uname -s | tr A-Z a-z`-`uname -m` all: build-it build-it: @test -e $(cmake_build_dir)/config.status || ./configure -@test -e $(cmake_build_dir)/CMakeCache.txt && \ test $(cmake_build_dir)/CMakeCache.txt -ot `cat $(cmake_build_dir)/CMakeCache.txt | grep ZEEK_DIST | cut -d '=' -f 2`/build/CMakeCache.txt && \ echo Updating stale CMake cache && \ touch $(cmake_build_dir)/CMakeCache.txt ( cd $(cmake_build_dir) && make ) install: ( cd $(cmake_build_dir) && make install ) clean: ( cd $(cmake_build_dir) && make clean ) distclean: rm -rf $(cmake_build_dir) test: make -C tests package-manager-3.0.1/testing/packages/rot13/zkg.meta0000644000175000017500000000017214565163601020501 0ustar rharha[package] script_dir = scripts/Demo/Rot13 build_command = ./configure && make test_command = cd testing && btest -d tests package-manager-3.0.1/testing/packages/rot13/configure0000755000175000017500000001232114565163601020744 0ustar rharha#!/bin/sh # # Wrapper for viewing/setting options that the plugin's CMake # scripts will recognize. # # This file is maintained by zkg. Do not edit. # Modify configure.plugin to add plugin-specific options. # set -e command="$0 $*" if [ -e `dirname $0`/configure.plugin ]; then # Include custom additions. . `dirname $0`/configure.plugin fi usage() { cat 1>&2 </dev/null 2>&1; then plugin_usage 1>&2 fi echo exit 1 } # Function to append a CMake cache entry definition to the # CMakeCacheEntries variable # $1 is the cache entry variable name # $2 is the cache entry variable type # $3 is the cache entry variable value append_cache_entry () { CMakeCacheEntries="$CMakeCacheEntries -D $1:$2=$3" } # set defaults builddir=build zeekdist="" installroot="default" CMakeCacheEntries="" while [ $# -ne 0 ]; do case "$1" in -*=*) optarg=`echo "$1" | sed 's/[-_a-zA-Z0-9]*=//'` ;; *) optarg= ;; esac case "$1" in --help|-h) usage ;; --cmake=*) CMakeCommand=$optarg ;; --zeek-dist=*) zeekdist=`cd $optarg && pwd` ;; --install-root=*) installroot=$optarg ;; --with-binpac=*) append_cache_entry BinPAC_ROOT_DIR PATH $optarg binpac_root=$optarg ;; --with-broker=*) append_cache_entry BROKER_ROOT_DIR PATH $optarg broker_root=$optarg ;; --with-caf=*) append_cache_entry CAF_ROOT_DIR PATH $optarg caf_root=$optarg ;; --with-bifcl=*) append_cache_entry BifCl_EXE PATH $optarg ;; --enable-debug) append_cache_entry BRO_PLUGIN_ENABLE_DEBUG BOOL true ;; *) if type plugin_option >/dev/null 2>&1; then plugin_option $1 && shift && continue; fi echo "Invalid option '$1'. Try $0 --help to see available options." exit 1 ;; esac shift done if [ -z "$CMakeCommand" ]; then # prefer cmake3 over "regular" cmake (cmake == cmake2 on RHEL) if command -v cmake3 >/dev/null 2>&1 ; then CMakeCommand="cmake3" elif command -v cmake >/dev/null 2>&1 ; then CMakeCommand="cmake" else echo "This package requires CMake, please install it first." echo "Then you may use this script to configure the CMake build." echo "Note: pass --cmake=PATH to use cmake in non-standard locations." exit 1; fi fi if [ -z "$zeekdist" ]; then if type zeek-config >/dev/null 2>&1; then zeek_config="zeek-config" else echo "Either 'zeek-config' must be in PATH or '--zeek-dist=' used" exit 1 fi append_cache_entry BRO_CONFIG_PREFIX PATH `${zeek_config} --prefix` append_cache_entry BRO_CONFIG_INCLUDE_DIR PATH `${zeek_config} --include_dir` append_cache_entry BRO_CONFIG_PLUGIN_DIR PATH `${zeek_config} --plugin_dir` append_cache_entry BRO_CONFIG_LIB_DIR PATH `${zeek_config} --lib_dir` append_cache_entry BRO_CONFIG_CMAKE_DIR PATH `${zeek_config} --cmake_dir` append_cache_entry CMAKE_MODULE_PATH PATH `${zeek_config} --cmake_dir` build_type=`${zeek_config} --build_type` if [ "$build_type" = "debug" ]; then append_cache_entry BRO_PLUGIN_ENABLE_DEBUG BOOL true fi if [ -z "$binpac_root" ]; then append_cache_entry BinPAC_ROOT_DIR PATH `${zeek_config} --binpac_root` fi if [ -z "$broker_root" ]; then append_cache_entry BROKER_ROOT_DIR PATH `${zeek_config} --broker_root` fi if [ -z "$caf_root" ]; then append_cache_entry CAF_ROOT_DIR PATH `${zeek_config} --caf_root` fi else if [ ! -e "$zeekdist/zeek-path-dev.in" ]; then echo "$zeekdist does not appear to be a valid Zeek source tree." exit 1 fi # BRO_DIST is the canonical/historical name used by plugin CMake scripts # ZEEK_DIST doesn't serve a function at the moment, but set/provided anyway append_cache_entry BRO_DIST PATH $zeekdist append_cache_entry ZEEK_DIST PATH $zeekdist append_cache_entry CMAKE_MODULE_PATH PATH $zeekdist/cmake fi if [ "$installroot" != "default" ]; then mkdir -p $installroot append_cache_entry BRO_PLUGIN_INSTALL_ROOT PATH $installroot fi if type plugin_addl >/dev/null 2>&1; then plugin_addl fi echo "Build Directory : $builddir" echo "Zeek Source Directory : $zeekdist" mkdir -p $builddir cd $builddir "$CMakeCommand" $CMakeCacheEntries .. echo "# This is the command used to configure this build" > config.status echo $command >> config.status chmod u+x config.status package-manager-3.0.1/testing/packages/rot13/CHANGES0000644000175000017500000000000014565163601020017 0ustar rharhapackage-manager-3.0.1/testing/packages/rot13/README0000644000175000017500000000012414565163601017713 0ustar rharha Demo::Rot13 ================================= package-manager-3.0.1/testing/packages/i-have-no-scripts/0000755000175000017500000000000014565163601021336 5ustar rharhapackage-manager-3.0.1/testing/packages/i-have-no-scripts/zkg.meta0000644000175000017500000000003114565163601022773 0ustar rharha[package] plugin_dir = . package-manager-3.0.1/testing/packages/grault/0000755000175000017500000000000014565163601017364 5ustar rharhapackage-manager-3.0.1/testing/packages/grault/zkg.meta0000644000175000017500000000001214565163601021020 0ustar rharha[package] package-manager-3.0.1/testing/packages/grault/__load__.zeek0000644000175000017500000000002714565163601021756 0ustar rharhaprint "grault loaded"; package-manager-3.0.1/testing/packages/quux/0000755000175000017500000000000014565163601017070 5ustar rharhapackage-manager-3.0.1/testing/packages/quux/zkg.meta0000644000175000017500000000001214565163601020524 0ustar rharha[package] package-manager-3.0.1/testing/packages/quux/__load__.zeek0000644000175000017500000000002514565163601021460 0ustar rharhaprint "quux loaded"; package-manager-3.0.1/testing/packages/corge/0000755000175000017500000000000014565163601017165 5ustar rharhapackage-manager-3.0.1/testing/packages/corge/zkg.meta0000644000175000017500000000001214565163601020621 0ustar rharha[package] package-manager-3.0.1/testing/packages/corge/__load__.zeek0000644000175000017500000000002614565163601021556 0ustar rharhaprint "corge loaded"; package-manager-3.0.1/testing/packages/bar/0000755000175000017500000000000014565163601016632 5ustar rharhapackage-manager-3.0.1/testing/packages/bar/zkg.meta0000644000175000017500000000001214565163601020266 0ustar rharha[package] package-manager-3.0.1/testing/packages/bar/__load__.zeek0000644000175000017500000000005214565163601021222 0ustar rharhaevent zeek_init() { print "bar loaded"; } package-manager-3.0.1/testing/packages/qux/0000755000175000017500000000000014565163601016703 5ustar rharhapackage-manager-3.0.1/testing/packages/qux/zkg.meta0000644000175000017500000000001214565163601020337 0ustar rharha[package] package-manager-3.0.1/testing/packages/qux/__load__.zeek0000644000175000017500000000002414565163601021272 0ustar rharhaprint "qux loaded"; package-manager-3.0.1/testing/.gitignore0000644000175000017500000000004014565163601016272 0ustar rharhadiag.log .btest.failed.dat .tmp package-manager-3.0.1/testing/scripts/0000755000175000017500000000000014565163601015777 5ustar rharhapackage-manager-3.0.1/testing/scripts/diff-remove-timestamps0000755000175000017500000000053114565163601022313 0ustar rharha#! /usr/bin/env bash # # Replace timestamps in the zkg log with XXXs. Note, this is different # from the canonifier used in the Zeek distribution. # Get us "modern" regexps with sed. if [ `uname` == "Linux" ]; then sed="sed -r" else sed="sed -E" fi $sed -e 's/^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}/XXXX-XX-XX XX:XX:XX/' package-manager-3.0.1/testing/scripts/diff-remove-cwd0000755000175000017500000000016314565163601020703 0ustar rharha#! /usr/bin/env bash # # Replace occurrences of the current working directory with "<...>" sed "s#$(pwd)#<...>#g" package-manager-3.0.1/testing/scripts/diff-remove-zkg-version0000755000175000017500000000027314565163601022406 0ustar rharha#! /usr/bin/env bash # # Suppress lines including our zkg version, based on the VERSION file # at the toplevel of the source tree. ver=$(cat $TEST_BASE/../VERSION) grep -v "$ver" exit 0 package-manager-3.0.1/testing/scripts/zkg-zeek0000755000175000017500000000112714565163601017455 0ustar rharha#! /usr/bin/env bash # # Invokes zkg to get the same behavior as with a Zeek-bundled install: # no config file is spelled out by default, but a config and state # directory are set as with the installation via CMake. # Pretend a different home directory, as created by setup-zeek-and-home export HOME=$(pwd)/home/testuser # Make zkg use the (pseudo) Zeek-bundled locations created by setup-zeek export ZEEK_ZKG_CONFIG_DIR=$(pwd)/zeekroot/etc/zkg export ZEEK_ZKG_STATE_DIR=$(pwd)/zeekroot/var/lib/zkg # Don't use a default package source in zkg: export ZKG_DEFAULT_SOURCE= $TEST_BASE/../zkg "$@" package-manager-3.0.1/testing/scripts/setup-zeek-and-home0000755000175000017500000000121214565163601021503 0ustar rharha#! /usr/bin/env bash # # Helper script that establishes a skeletal equivalent of a # Zeek-bundled zkg state & config, and provides an alternative home # directory. zeek_zkg_config_dir=$(pwd)/zeekroot/etc/zkg zeek_zkg_state_dir=$(pwd)/zeekroot/var/lib/zkg zeek_site_dir=$(pwd)/zeekroot/share/zeek/site zeek_plugins_dir=$(pwd)/zeekroot/lib/zeek/plugins mkdir -p home/testuser ${zeek_zkg_config_dir} ${zeek_zkg_state_dir} \ ${zeek_site_dir} ${zeek_plugins_dir} echo "\ [sources] one = $(pwd)/sources/one [paths] state_dir = ${zeek_zkg_state_dir} script_dir = ${zeek_site_dir} plugin_dir = ${zeek_plugins_dir} " >> ${zeek_zkg_config_dir}/config package-manager-3.0.1/testing/scripts/zkg0000755000175000017500000000062514565163601016523 0ustar rharha#! /usr/bin/env bash force="" case "$1" in install) force="--force" ;; upgrade) force="--force" ;; bundle) force="--force" ;; unbundle) force="--force" ;; remove) force="--force" ;; unload) force="--force" ;; purge) force="--force" ;; esac $TEST_BASE/../zkg --configfile=config "$@" $force package-manager-3.0.1/testing/scripts/initializer0000755000175000017500000000317714565163601020260 0ustar rharha#! /usr/bin/env bash # # This initializer runs prior to every test. It creates a fresh copy of our # test packages and package sources. Individual tests may tweak these further as # needed, prior to running tests with them. The initializer also creates a zkg # config that all tests automatically run with, via the "zkg" wrapper script in # this directory. cp -R $PACKAGES . for p in packages/*; do if [ $p = 'packages/foo' ]; then # Use a different default branch than 'master' for testing purposes ( cd $p && git init && git checkout -b main && git add * && git commit -m 'init' ) else ( cd $p && git init && git add * && git commit -m 'init' ) fi done cp -R $SOURCES . find sources -name 'zkg.index' -exec sed -i -e "s#^#$(pwd)/packages/#" {} \; for s in sources/*; do ( cd $s && git init && git add * && git commit -m 'init' ) done # Create a branch drop-corge in source "one" to drop corge from index ( cd sources/one && default_branch=$(git rev-parse --abbrev-ref HEAD) && git checkout -b drop-corge && sed -i -e '/corge/d' bob/zkg.index && git add ./bob/zkg.index && git commit -m 'Remove corge' && git checkout ${default_branch} ) cp -R $TEMPLATES . for t in templates/*; do ( cd $t && git init && git add * && git commit -m 'init' git remote add origin https://example.com/zeek/package-template ) done echo "\ [sources] one = $(pwd)/sources/one [paths] state_dir = $(pwd)/state script_dir = $(pwd)/scripts plugin_dir = $(pwd)/plugins bin_dir = $(pwd)/bin " >> config type zeek-config > /dev/null 2>&1 && echo "zeek_dist = $(zeek-config --zeek_dist)" >> config || true package-manager-3.0.1/testing/scripts/diff-canonifier0000755000175000017500000000037114565163601020751 0ustar rharha#! /usr/bin/env bash # # Default canonifier that combines several others. DIR="$(cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd)" $DIR/diff-remove-timestamps \ | $DIR/diff-remove-zkg-version \ | $DIR/diff-remove-abspath exit 0 package-manager-3.0.1/testing/scripts/diff-remove-zkg-meta-commit0000755000175000017500000000040014565163601023125 0ustar rharha#! /usr/bin/env bash # # Suppress the SHA commit string in a zkg.meta's template.commit line. # Get us "modern" regexps with sed. if [ `uname` == "Linux" ]; then sed="sed -r" else sed="sed -E" fi $sed -e 's/commit = [0-9a-z]{8}/commit = xxxxxxxx/' package-manager-3.0.1/testing/scripts/git0000755000175000017500000000064414565163601016514 0ustar rharha#! /usr/bin/env bash # Git wrapper script for use during testing. # Use original path so we find the system's installed git, not this wrapper. PATH="$ORIGPATH" # Unsetting the following prevents git from reading ~/.gitconfig, # including potential githooks. HOME= XDG_CONFIG_HOME= git -c init.defaultBranch=master \ -c protocol.file.allow=always \ -c user.name=zkg \ -c user.email=zkg@zeek.org \ "$@" package-manager-3.0.1/testing/scripts/diff-remove-abspath0000755000175000017500000000032614565163601021551 0ustar rharha#! /usr/bin/env bash # # Replace absolute paths with the basename. if [ `uname` == "Linux" ]; then sed="sed -r" else sed="sed -E" fi $sed 's#/+#/#g' | \ $sed 's#/([^ :/]{1,}/){1,}([^ :/]{1,})#<...>/\2#g' package-manager-3.0.1/testing/baselines/0000755000175000017500000000000014565163601016255 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.metadata-aliases/0000755000175000017500000000000014565163601022615 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.metadata-aliases/scripts.foo3.__load__.zeek0000644000175000017500000000020714565163601027542 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo loaded"; package-manager-3.0.1/testing/baselines/tests.metadata-aliases/scripts.foo.__load__.zeek0000644000175000017500000000020714565163601027457 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo loaded"; package-manager-3.0.1/testing/baselines/tests.metadata-aliases/scripts.foo2.__load__.zeek0000644000175000017500000000020714565163601027541 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo loaded"; package-manager-3.0.1/testing/baselines/tests.refresh/0000755000175000017500000000000014565163601021054 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.refresh/agg.fail.out0000644000175000017500000000057314565163601023262 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Refresh package source: one No membership changes WARNING: Metadata aggregated, but excludes the following packages due to described problems: <...>/bad_pkg: missing zkg.meta (or bro-pkg.meta) metadata file Refresh installed packages No new outdated packages package-manager-3.0.1/testing/baselines/tests.refresh/agg.errout0000644000175000017500000000140314565163601023052 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. XXXX-XX-XX XX:XX:XX DEBUG found source clone of "one" at <...>/one XXXX-XX-XX XX:XX:XX DEBUG refresh "one": pulling <...>/one XXXX-XX-XX XX:XX:XX WARNING <...>/bro-pkg.meta: missing metadata file XXXX-XX-XX XX:XX:XX WARNING skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file XXXX-XX-XX XX:XX:XX DEBUG metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals XXXX-XX-XX XX:XX:XX INFO committed package source "one" metadata update XXXX-XX-XX XX:XX:XX DEBUG fetch package one<...>/foo package-manager-3.0.1/testing/baselines/tests.refresh/list.out0000644000175000017500000000043214565163601022557 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bad_pkg one/alice/bar one/alice/baz one/alice/foo (installed: main) one/alice/i-have-no-scripts one/alice/new_pkg one/alice/qux one/bob/corge one/bob/grault package-manager-3.0.1/testing/baselines/tests.refresh/agg.out0000644000175000017500000000062714565163601022350 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Refresh package source: one No membership changes WARNING: Metadata aggregated, but excludes the following packages due to described problems: <...>/bad_pkg: missing zkg.meta (or bro-pkg.meta) metadata file Pushed aggregated metadata Refresh installed packages No new outdated packages package-manager-3.0.1/testing/baselines/tests.refresh/list_after_agg.out0000644000175000017500000000050414565163601024556 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bad_pkg one/alice/bar one/alice/baz one/alice/foo (installed: main) one/alice/i-have-no-scripts one/alice/new_pkg - This is the new_pkg package description one/alice/qux one/bob/corge one/bob/grault package-manager-3.0.1/testing/baselines/tests.refresh/outdated.out0000644000175000017500000000022314565163601023413 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.refresh/agg.fail.errout0000644000175000017500000000126714565163601023774 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. XXXX-XX-XX XX:XX:XX DEBUG found source clone of "one" at <...>/one XXXX-XX-XX XX:XX:XX DEBUG refresh "one": pulling <...>/one XXXX-XX-XX XX:XX:XX WARNING <...>/bro-pkg.meta: missing metadata file XXXX-XX-XX XX:XX:XX WARNING skipping aggregation of <...>/bad_pkg: bad metadata: missing zkg.meta (or bro-pkg.meta) metadata file XXXX-XX-XX XX:XX:XX DEBUG metadata refresh: 8 additions (alice/bar, alice/baz, alice/foo, alice/i-have-no-scripts, alice/new_pkg, alice/qux, bob/corge, bob/grault), 0 changes, 0 removals XXXX-XX-XX XX:XX:XX DEBUG fetch package one<...>/foo package-manager-3.0.1/testing/baselines/tests.refresh/search.out0000644000175000017500000000036514565163601023056 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) - This is the foo package description one/alice/new_pkg - This is the new_pkg package description package-manager-3.0.1/testing/baselines/tests.builtin-spicy-bundle/0000755000175000017500000000000014565163601023460 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-spicy-bundle/package.log0000644000175000017500000000024614565163601025560 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Building foo Testing foo Building foo Building foo package-manager-3.0.1/testing/baselines/tests.builtin-spicy-bundle/.stderr0000644000175000017500000000032514565163601024764 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Switched to a new branch 'main' Switched to a new branch 'drop-corge' Switched to branch 'master' package-manager-3.0.1/testing/baselines/tests.builtin-spicy-bundle/bundle.manifest.txt0000644000175000017500000000027414565163601027302 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. [bundle] <...>/foo = main [meta] builtin_packages = spicy-plugin=X.X.X package-manager-3.0.1/testing/baselines/tests.builtin-spicy-bundle/output0000644000175000017500000000050114565163601024737 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Running unit tests for "one/alice/foo" Installing "one/alice/foo" Installed "one/alice/foo" (main) Loaded "one/alice/foo" Bundle successfully written: bundle.tar Loaded "one/alice/foo" Unbundling complete. package-manager-3.0.1/testing/baselines/tests.load/0000755000175000017500000000000014565163601020335 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.load/after_unload.out0000644000175000017500000000032014565163601023524 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./bar package-manager-3.0.1/testing/baselines/tests.load/after_load.out0000644000175000017500000000033414565163601023166 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./bar @load ./foo package-manager-3.0.1/testing/baselines/tests.load/after_loading_no_scripts.out0000644000175000017500000000033414565163601026127 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./bar @load ./foo package-manager-3.0.1/testing/baselines/tests.load/scripts.packages.packages.zeek0000644000175000017500000000033414565163601026236 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./bar @load ./foo package-manager-3.0.1/testing/baselines/tests.bindir-relocate/0000755000175000017500000000000014565163601022461 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.bindir-relocate/output0000644000175000017500000000021114565163601023736 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. from exec1 from exec2 package-manager-3.0.1/testing/baselines/tests.dependency-management/0000755000175000017500000000000014565163601023646 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.dependency-management/out0000644000175000017500000000162614565163601024405 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. ** Install grault Installing "one/bob/corge" Installed "one/bob/corge" (1.0.0) Loaded "one/bob/corge" Installing "one/bob/grault" Installed "one/bob/grault" (1.0.0) Loaded "one/bob/grault" ** Unload grault Unloaded "one/bob/grault" ** Install foo Installing "one/alice/baz" Installed "one/alice/baz" (1.0.0) Loaded "one/alice/baz" Installing "one/alice/bar" Installed "one/alice/bar" (1.0.0) Loaded "one/alice/bar" Installing "one/alice/foo" Installed "one/alice/foo" (main) Loaded "one/alice/foo" The following installed packages were additionally loaded to satisfy runtime dependencies grault ** Unload foo Unloaded "one/alice/foo" ** Load foo Loaded "one/alice/foo" ** Unload bar Unloaded "one/alice/bar" Unloaded "one/alice/foo" ** Remove grault Unloaded "one/alice/baz" Removed "one/bob/grault" package-manager-3.0.1/testing/baselines/tests.template-info/0000755000175000017500000000000014565163601022162 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.template-info/output.plain0000644000175000017500000000056314565163601024553 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. API version: 1.0.0 features: readme origin: not a git repository provides package: true user vars: name: the name of the package, e.g. "FooBar", no default, used by package readme: Content of the README file, This is a README., used by readme versions: package-manager-3.0.1/testing/baselines/tests.template-info/output.git0000644000175000017500000000133614565163601024232 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. { "api_version": "1.0.0", "features": [ "readme" ], "has_repo": true, "origin": "https://example.com/zeek/package-template", "provides_package": true, "user_vars": { "name": { "default": null, "description": "the name of the package, e.g. \"FooBar\"", "used_by": [ "package" ] }, "readme": { "default": "This is a README.", "description": "Content of the README file", "used_by": [ "readme" ] } }, "versions": [] } package-manager-3.0.1/testing/baselines/tests.template-info/output.json0000644000175000017500000000131214565163601024412 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. { "api_version": "1.0.0", "features": [ "readme" ], "has_repo": false, "origin": "not a git repository", "provides_package": true, "user_vars": { "name": { "default": null, "description": "the name of the package, e.g. \"FooBar\"", "used_by": [ "package" ] }, "readme": { "default": "This is a README.", "description": "Content of the README file", "used_by": [ "readme" ] } }, "versions": [] } package-manager-3.0.1/testing/baselines/tests.builtin-test/0000755000175000017500000000000014565163601022041 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-test/out0000644000175000017500000000025114565163601022571 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. cannot run tests for "spicy-plugin": built-in package package-manager-3.0.1/testing/baselines/tests.builtin-spicy/0000755000175000017500000000000014565163601022211 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-spicy/package.log0000644000175000017500000000023114565163601024303 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Building foo Testing foo Building foo package-manager-3.0.1/testing/baselines/tests.builtin-spicy/output0000644000175000017500000000017214565163601023474 0ustar rharhaRunning unit tests for "one/alice/foo" Installing "one/alice/foo" Installed "one/alice/foo" (main) Loaded "one/alice/foo" package-manager-3.0.1/testing/baselines/tests.aliases-bad/0000755000175000017500000000000014565163601021563 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.aliases-bad/.stderr0000644000175000017500000000027514565163601023073 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: invalid package "foo": invalid alias "../../../../../../../../bad" package-manager-3.0.1/testing/baselines/tests.aliases-bad-2/0000755000175000017500000000000014565163601021722 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.aliases-bad-2/.stderr0000644000175000017500000000025114565163601023224 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: invalid package "foo": invalid alias ".hidden" package-manager-3.0.1/testing/baselines/tests.metadata-depends/0000755000175000017500000000000014565163601022616 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.metadata-depends/upgrade_depends.out0000644000175000017500000000043114565163601026476 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: main) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-3.0.1/testing/baselines/tests.metadata-depends/install_depends.out0000644000175000017500000000043114565163601026515 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: main) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-3.0.1/testing/baselines/tests.metadata-depends/no_depends.out0000644000175000017500000000022314565163601025462 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.metadata-depends/bundle_depends.out0000644000175000017500000000043114565163601026320 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: main) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-3.0.1/testing/baselines/tests.builtin-unpin/0000755000175000017500000000000014565163601022213 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-unpin/out0000644000175000017500000000024114565163601022742 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. cannot unpin "spicy-plugin": built-in package package-manager-3.0.1/testing/baselines/tests.branch-based-dependency/0000755000175000017500000000000014565163601024043 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.branch-based-dependency/installed.out0000644000175000017500000000030114565163601026545 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: release/1.0) one/alice/foo (installed: release/1.0) package-manager-3.0.1/testing/baselines/tests.builtin-install/0000755000175000017500000000000014565163601022530 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-install/out0000644000175000017500000000024314565163601023261 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. cannot install "spicy-plugin": built-in package package-manager-3.0.1/testing/baselines/tests.remove/0000755000175000017500000000000014565163601020713 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.remove/scripts.packages.packages.zeek0000644000175000017500000000033414565163601026614 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./bar @load ./baz package-manager-3.0.1/testing/baselines/tests.install-invalid/0000755000175000017500000000000014565163601022510 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.install-invalid/output0000644000175000017500000000067514565163601024003 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: invalid package ".": Package name ' fronting-whitespace' is not valid. error: invalid package ".": Package name 'trailing-whitespace ' is not valid. error: path ./packages/doesntexist is not a git repository error: path ./packages/notagitrepo is not a git repository error: local git clone at ./packages/dirtyrepo is dirty package-manager-3.0.1/testing/baselines/tests.install-force-test-fail/0000755000175000017500000000000014565163601024046 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.install-force-test-fail/out0000644000175000017500000000062414565163601024602 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. === install without skiptests error: failed to run tests for one<...>/foo: test_command failed with exit code 1 Running unit tests for "one<...>/foo" have_load_foo=0 === install with skiptests Installing "one<...>/foo" Installed "one<...>/foo" (main) Loaded "one<...>/foo" have_load_foo=1 package-manager-3.0.1/testing/baselines/tests.builtin-spicy-version/0000755000175000017500000000000014565163601023674 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-spicy-version/package.log0000644000175000017500000000023114565163601025766 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Building foo Testing foo Building foo package-manager-3.0.1/testing/baselines/tests.builtin-spicy-version/output0000644000175000017500000000035514565163601025162 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Running unit tests for "one/alice/foo" Installing "one/alice/foo" Installed "one/alice/foo" (main) Loaded "one/alice/foo" package-manager-3.0.1/testing/baselines/tests.bundle-missing-dependency/0000755000175000017500000000000014565163601024452 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.bundle-missing-dependency/package.log0000644000175000017500000000020014565163601026540 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Building foo package-manager-3.0.1/testing/baselines/tests.bundle-missing-dependency/.stderr0000644000175000017500000000052514565163601025760 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Switched to a new branch 'main' Switched to a new branch 'drop-corge' Switched to branch 'master' Switched to a new branch 'origin/main' XXXX-XX-XX XX:XX:XX WARNING dependency "zkg-test-plugin" of bundled "<...>/foo" missing package-manager-3.0.1/testing/baselines/tests.bundle-missing-dependency/output0000644000175000017500000000030114565163601025727 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Loaded "/one/alice/foo" Unbundling complete. /one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.builtin-pin/0000755000175000017500000000000014565163601021650 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-pin/out0000644000175000017500000000023714565163601022404 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. cannot pin "spicy-plugin": built-in package package-manager-3.0.1/testing/baselines/tests.builtin-spicy-version-error/0000755000175000017500000000000014565163601025023 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-spicy-version-error/output0000644000175000017500000000044014565163601026304 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: failed to resolve dependencies: unsatisfiable dependency: "zeek-builtin/spicy-plugin" (X.X.X) is installed, but "one/alice/foo" requires = 0.63. cannot remove "spicy-plugin": built-in package package-manager-3.0.1/testing/baselines/tests.purge/0000755000175000017500000000000014565163601020540 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.purge/scripts.packages.packages.zeek0000644000175000017500000000030414565163601026436 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. package-manager-3.0.1/testing/baselines/tests.list/0000755000175000017500000000000014565163601020371 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.list/installed.out0000644000175000017500000000026514565163601023104 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: master) one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.list/all.out0000644000175000017500000000034414565163601021673 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar one/alice/baz one/alice/foo one/alice/i-have-no-scripts one/alice/qux one/bob/corge one/bob/grault package-manager-3.0.1/testing/baselines/tests.list/not_installed.out0000644000175000017500000000031014565163601023753 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/baz one/alice/i-have-no-scripts one/alice/qux one/bob/corge one/bob/grault package-manager-3.0.1/testing/baselines/tests.list/unloaded.out0000644000175000017500000000022514565163601022714 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: master) package-manager-3.0.1/testing/baselines/tests.list/outdated.out0000644000175000017500000000022314565163601022730 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.list/after_upgrade.out0000644000175000017500000000016314565163601023732 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. package-manager-3.0.1/testing/baselines/tests.list/loaded.out0000644000175000017500000000022314565163601022347 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.package-submodule/0000755000175000017500000000000014565163601023006 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.package-submodule/foo.install0000644000175000017500000000023514565163601025161 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. event zeek_init() { print "bar loaded"; } package-manager-3.0.1/testing/baselines/tests.package-submodule/foo.upgrade0000644000175000017500000000025014565163601025137 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. event zeek_init() { print "bar upgrade is loaded"; } package-manager-3.0.1/testing/baselines/tests.aliases-conflict/0000755000175000017500000000000014565163601022636 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.aliases-conflict/out0000644000175000017500000000240714565163601023373 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Installing "one/alice/foo" Installed "one/alice/foo" (main) Loaded "one/alice/foo" Installing "one/alice/bar" error: incomplete installation, the follow packages failed to be installed: one/alice/bar (master) Failed installing "one/alice/bar": name "bar" conflicts with alias from "one/alice/foo" Installing "one/alice/baz" error: incomplete installation, the follow packages failed to be installed: one/alice/baz (master) Failed installing "one/alice/baz": alias "foo" conflicts with name of installed package "one/alice/foo" Installing "one/bob/corge" error: incomplete installation, the follow packages failed to be installed: one/bob/corge (master) Failed installing "one/bob/corge": alias "bar" conflicts with alias of installed package "one/alice/foo" Removed "one/alice/foo" Installing "one/alice/baz" Installed "one/alice/baz" (master) Loaded "one/alice/baz" Installing "one/bob/corge" Installed "one/bob/corge" (master) Loaded "one/bob/corge" Installing "one/alice/foo" error: incomplete installation, the follow packages failed to be installed: one/alice/foo (main) Failed installing "one/alice/foo": name "foo" conflicts with alias from "one/alice/baz" package-manager-3.0.1/testing/baselines/tests.aliases-conflict/packages.zeek.20000644000175000017500000000033614565163601025436 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./baz @load ./corge package-manager-3.0.1/testing/baselines/tests.aliases-conflict/packages.zeek.10000644000175000017500000000032014565163601025426 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./foo package-manager-3.0.1/testing/baselines/tests.upgrade-test-fail/0000755000175000017500000000000014565163601022733 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.upgrade-test-fail/out0000644000175000017500000000064714565163601023474 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Installing "one<...>/foo" Installed "one<...>/foo" (1.0.2) Loaded "one<...>/foo" === upgrade without skiptests error: failed to run tests for one<...>/foo: test_command failed with exit code 1 Running unit tests for "one<...>/foo" === upgrade with skiptest Upgraded "one<...>/foo" (1.0.3) print "foo 1.0.3"; package-manager-3.0.1/testing/baselines/tests.bundled-zkg/0000755000175000017500000000000014565163601021624 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.bundled-zkg/output0000644000175000017500000000024714565163601023112 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. /tests.bundled-zkg/install-dir/bin/zkg zkg xxx package-manager-3.0.1/testing/baselines/tests.uninstall/0000755000175000017500000000000014565163601021427 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.uninstall/scripts.packages.packages.zeek0000644000175000017500000000033414565163601027330 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # WARNING: This file is managed by zkg. # Do not make direct modifications here. @load ./bar @load ./baz package-manager-3.0.1/testing/baselines/tests.metadata-script_dir/0000755000175000017500000000000014565163601023336 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.metadata-script_dir/scripts.foo.__load__.zeek0000644000175000017500000000020714565163601030200 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo loaded"; package-manager-3.0.1/testing/baselines/tests.metadata-suggests/0000755000175000017500000000000014565163601023040 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.metadata-suggests/no_suggests.out0000644000175000017500000000022314565163601026126 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.metadata-suggests/bundle_suggests.out0000644000175000017500000000043114565163601026764 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: main) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-3.0.1/testing/baselines/tests.metadata-suggests/install_suggests.out0000644000175000017500000000043114565163601027161 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: main) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-3.0.1/testing/baselines/tests.metadata-suggests/upgrade_suggests.out0000644000175000017500000000043114565163601027142 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: 2.0.0) one/alice/foo (installed: main) one/bob/corge (installed: 1.0.1) one/bob/grault (installed: master) package-manager-3.0.1/testing/baselines/tests.install-reserved/0000755000175000017500000000000014565163601022701 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.install-reserved/output0000644000175000017500000000026514565163601024167 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: invalid package ".": Package name 'packages' is not valid. package-manager-3.0.1/testing/baselines/tests.bundle-unsatisfied-dependency/0000755000175000017500000000000014565163601025317 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.bundle-unsatisfied-dependency/package.log0000644000175000017500000000020014565163601027405 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Building foo package-manager-3.0.1/testing/baselines/tests.bundle-unsatisfied-dependency/.stderr0000644000175000017500000000054714565163601026631 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Switched to a new branch 'main' Switched to a new branch 'drop-corge' Switched to branch 'master' Switched to a new branch 'origin/main' XXXX-XX-XX XX:XX:XX WARNING dependency "spicy-plugin" (X.X.X) of "<...>/foo" not compatible with "<6.0.0" package-manager-3.0.1/testing/baselines/tests.bundle-unsatisfied-dependency/output0000644000175000017500000000030114565163601026574 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Loaded "/one/alice/foo" Unbundling complete. /one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.unsatisfied-version-tag-dependency/0000755000175000017500000000000014565163601026304 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.unsatisfied-version-tag-dependency/fail.out0000644000175000017500000000036714565163601027756 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: failed to resolve dependencies: "one/alice/bar" has no version satisfying dependencies: "one/alice/foo" requires: "=1.0.0" package-manager-3.0.1/testing/baselines/tests.aliases-bad-3/0000755000175000017500000000000014565163601021723 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.aliases-bad-3/.stderr0000644000175000017500000000024614565163601023231 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: invalid package "foo": invalid alias "/aaa" package-manager-3.0.1/testing/baselines/tests.load-zeek-dependency/0000755000175000017500000000000014565163601023405 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.load-zeek-dependency/nothing-loaded.out0000644000175000017500000000016314565163601027032 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. package-manager-3.0.1/testing/baselines/tests.load-zeek-dependency/foo-loaded.out0000644000175000017500000000022314565163601026144 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/foo (installed: main) package-manager-3.0.1/testing/baselines/tests.dependency-ordering/0000755000175000017500000000000014565163601023343 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.dependency-ordering/build.log0000644000175000017500000000037714565163601025154 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. building grault building corge building baz building bar building foo building grault building corge building baz building bar building foo package-manager-3.0.1/testing/baselines/tests.builtin-info/0000755000175000017500000000000014565163601022015 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-info/out0000644000175000017500000000063314565163601022551 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. "zeek-builtin/spicy-plugin" info: url: zeek-builtin://spicy-plugin versions: ['X.X.X'] install status: current_hash = None current_version = X.X.X is_loaded = True is_outdated = False is_pinned = True tracking_method = builtin metadata (from version ""): package-manager-3.0.1/testing/baselines/tests.install-bro-pkg-warning/0000755000175000017500000000000014565163601024066 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.install-bro-pkg-warning/stderr0000644000175000017500000000055014565163601025314 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. XXXX-XX-XX XX:XX:XX WARNING Package one/alice/foo is using the legacy bro-pkg.meta metadata file. While bro-pkg.meta still functions, it is recommended to use zkg.meta instead for future-proofing. Please report this to the package maintainers. package-manager-3.0.1/testing/baselines/tests.info/0000755000175000017500000000000014565163601020351 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.info/bar.info0000644000175000017500000000062414565163601021774 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. "one<...>/bar" info: versions: ['1.0.0'] install status: current_version = 1.0.0 is_loaded = True is_outdated = False is_pinned = False tracking_method = version metadata file: <...>/zkg.meta metadata (from version "1.0.0"): description = bar bar bar tags = sega sunset package-manager-3.0.1/testing/baselines/tests.info/foo.info0000644000175000017500000000061114565163601022007 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. "one<...>/foo" info: versions: ['1.0.0', '1.0.1', '1.0.2'] install status: current_version = main is_loaded = True is_outdated = True is_pinned = False tracking_method = branch metadata file: <...>/zkg.meta metadata (from version "main"): package-manager-3.0.1/testing/baselines/tests.info/installed.info0000644000175000017500000000061114565163601023203 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. "one<...>/foo" info: versions: ['1.0.0', '1.0.1', '1.0.2'] install status: current_version = main is_loaded = True is_outdated = True is_pinned = False tracking_method = branch metadata file: <...>/zkg.meta metadata (from version "main"): package-manager-3.0.1/testing/baselines/tests.template-create/0000755000175000017500000000000014565163601022472 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.template-create/out3.README0000644000175000017500000000024014565163601024237 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. This is the test3 package. This is a README. package-manager-3.0.1/testing/baselines/tests.template-create/out3.scripts.main.zeek0000644000175000017500000000026114565163601026654 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. module TEST3; event zeek_init() { print "Hello world!"; } package-manager-3.0.1/testing/baselines/tests.template-create/output0000644000175000017500000000162714565163601023763 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. "foo" will use value of "name" (the name of the package, e.g. "FooBar") from command line: test1 "foo" will use value of "name" (the name of the package, e.g. "FooBar") from environment: test2 "foo" will use value of "name" (the name of the package, e.g. "FooBar") from command line: test3 "foo" will use value of "name" (the name of the package, e.g. "FooBar") from command line: test4 <...>/out1 (installed: master) - TODO: A more detailed description of test1. <...>/out2 (installed: master) - TODO: A more detailed description of test2. <...>/out3 (installed: master) - TODO: A more detailed description of test3. error: could not determine value of user variable "name", provide via environment or --user-var error: the following features are unknown: "doesntexist". Template "foo" offers "readme". package-manager-3.0.1/testing/baselines/tests.template-create/out4.zkg.meta0000644000175000017500000000066014565163601025031 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. [package] script_dir = scripts summary = TODO: A summary of test4 in one line description = TODO: A more detailed description of test4. It can span multiple lines, with this indentation. [template] source = https://example.com/zeek/package-template version = master commit = xxxxxxxx [template_vars] name = test4 package-manager-3.0.1/testing/baselines/tests.template-create/out3.zkg.meta0000644000175000017500000000065214565163601025031 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. [package] script_dir = scripts summary = TODO: A summary of test3 in one line description = TODO: A more detailed description of test3. It can span multiple lines, with this indentation. [template] source = foo version = unversioned features = readme [template_vars] name = test3 readme = This is a README. package-manager-3.0.1/testing/baselines/tests.metadata-config_files/0000755000175000017500000000000014565163601023623 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.metadata-config_files/scripts.bar.config.zeek0000644000175000017500000000020414565163601030175 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "bar mod"; package-manager-3.0.1/testing/baselines/tests.metadata-config_files/bar_backup.zeek0000644000175000017500000000020414565163601026570 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "bar mod"; package-manager-3.0.1/testing/baselines/tests.metadata-config_files/barconfig2.zeek0000644000175000017500000000020614565163601026515 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "bar 2.0.0"; package-manager-3.0.1/testing/baselines/tests.metadata-config_files/foo_backup.zeek0000644000175000017500000000020414565163601026607 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo mod"; package-manager-3.0.1/testing/baselines/tests.metadata-config_files/scripts.foo.config.zeek0000644000175000017500000000020414565163601030214 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo new"; package-manager-3.0.1/testing/baselines/tests.install/0000755000175000017500000000000014565163601021064 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.install/scripts.packages.baz.__load__.zeek0000644000175000017500000000020714565163601027474 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "baz loaded"; package-manager-3.0.1/testing/baselines/tests.install/scripts.packages.qux.__load__.zeek0000644000175000017500000000020714565163601027535 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "qux loaded"; package-manager-3.0.1/testing/baselines/tests.install/scripts.packages.corge.__load__.zeek0000644000175000017500000000021114565163601030012 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "corge loaded"; package-manager-3.0.1/testing/baselines/tests.install/scripts.packages.grault.__load__.zeek0000644000175000017500000000021114565163601030211 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "grault 1.0.2"; package-manager-3.0.1/testing/baselines/tests.install/grault.detached0000644000175000017500000000021114565163601024037 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "grault 1.0.2"; package-manager-3.0.1/testing/baselines/tests.install/grault.detached.head0000644000175000017500000000021114565163601024737 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "grault 1.0.0"; package-manager-3.0.1/testing/baselines/tests.install/grault.info0000644000175000017500000000021614565163601023236 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. tracking_method = commit package-manager-3.0.1/testing/baselines/tests.install/corge.master0000644000175000017500000000023014565163601023373 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "corge loaded"; print "hello"; package-manager-3.0.1/testing/baselines/tests.install/grault.1.0.10000644000175000017500000000021114565163601022733 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "grault 1.0.1"; package-manager-3.0.1/testing/baselines/tests.install/scripts.packages.foo.__load__.zeek0000644000175000017500000000020714565163601027503 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo loaded"; package-manager-3.0.1/testing/baselines/tests.install/scripts.packages.bar.__load__.zeek0000644000175000017500000000023514565163601027465 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. event zeek_init() { print "bar loaded"; } package-manager-3.0.1/testing/baselines/tests.package_base/0000755000175000017500000000000014565163601022003 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.package_base/state.testing.foo.clones.test.log0000644000175000017500000000022214565163601030320 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. <...>/state/testing/foo/clones package-manager-3.0.1/testing/baselines/tests.package_base/state.logs.foo-build.log0000644000175000017500000000022514565163601026447 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. === STDERR === === STDOUT === foo package-manager-3.0.1/testing/baselines/tests.bundle/0000755000175000017500000000000014565163601020667 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.bundle/snapshot.out0000644000175000017500000000032714565163601023261 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/baz (installed: master) one/alice/foo (installed: 1.0.2) package-manager-3.0.1/testing/baselines/tests.bundle/manifest.out0000644000175000017500000000032714565163601023230 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.1) one/alice/baz (installed: master) one/alice/foo (installed: 1.0.0) package-manager-3.0.1/testing/baselines/tests.bundle/args.out0000644000175000017500000000032714565163601022356 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.2) one/alice/baz (installed: master) one/alice/foo (installed: 1.0.2) package-manager-3.0.1/testing/baselines/tests.builtin-unload/0000755000175000017500000000000014565163601022344 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-unload/out0000644000175000017500000000024214565163601023074 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. cannot unload "spicy-plugin": built-in package package-manager-3.0.1/testing/baselines/tests.upgrade/0000755000175000017500000000000014565163601021045 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.upgrade/scripts.packages.foo.__load__.zeek0000644000175000017500000000020614565163601027463 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo 1.0.4"; package-manager-3.0.1/testing/baselines/tests.upgrade/scripts.packages.bar.__load__.zeek0000644000175000017500000000021114565163601027440 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "bar++ loaded"; package-manager-3.0.1/testing/baselines/tests.install-force-build-fail/0000755000175000017500000000000014565163601024166 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.install-force-build-fail/out0000644000175000017500000000055614565163601024726 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Installing "one<...>/foo" error: incomplete installation, the follow packages failed to be installed: one<...>/foo (main) Failed installing "one<...>/foo": package build_command failed, see log in <...>/foo-build.log have_load_bar=1 have_load_foo=0 package-manager-3.0.1/testing/baselines/tests.installed-dependency-conflict/0000755000175000017500000000000014565163601025310 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.installed-dependency-conflict/installed-initial.out0000644000175000017500000000026514565163601031452 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/foo (installed: 1.0.0) package-manager-3.0.1/testing/baselines/tests.installed-dependency-conflict/installed-final.out0000644000175000017500000000026514565163601031112 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. one/alice/bar (installed: 1.0.0) one/alice/foo (installed: 1.0.0) package-manager-3.0.1/testing/baselines/tests.installed-dependency-conflict/conflict.out0000644000175000017500000000042414565163601027642 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. error: failed to resolve dependencies: unsatisfiable dependency: "one/alice/bar" (1.0.0) is installed, but "one/alice/foo" requires =2.0.0 (1.0.0 not in =2.0.0) package-manager-3.0.1/testing/baselines/tests.bindir-install/0000755000175000017500000000000014565163601022331 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.bindir-install/output0000644000175000017500000000021114565163601023606 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. from exec1 from exec2 package-manager-3.0.1/testing/baselines/tests.builtin-list/0000755000175000017500000000000014565163601022035 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.builtin-list/out0000644000175000017500000000024014565163601022563 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. zeek-builtin/spicy-plugin (installed: X.X.X) package-manager-3.0.1/testing/baselines/tests.pin/0000755000175000017500000000000014565163601020204 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.pin/foo_after_unpin.out0000644000175000017500000000020614565163601024110 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo 1.0.4"; package-manager-3.0.1/testing/baselines/tests.pin/scripts.packages.foo.__load__.zeek0000644000175000017500000000020614565163601026622 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "foo 1.0.2"; package-manager-3.0.1/testing/baselines/tests.pin/bar_after_unpin.out0000644000175000017500000000021114565163601024065 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. print "bar++ loaded"; package-manager-3.0.1/testing/baselines/tests.pin/scripts.packages.bar.__load__.zeek0000644000175000017500000000023514565163601026605 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. event zeek_init() { print "bar loaded"; } package-manager-3.0.1/testing/baselines/tests.validate-manifest/0000755000175000017500000000000014565163601023013 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.validate-manifest/packages.txt0000644000175000017500000000024314565163601025331 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. 0 alice foo one 1 alice bar one 2 alice baz one package-manager-3.0.1/testing/baselines/tests.plugin/0000755000175000017500000000000014565163601020714 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.plugin/scripts.packages.rot13.__load__.zeek0000644000175000017500000000045514565163601027525 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # # This is loaded when a user activates the plugin. Include scripts here that should be # loaded automatically at that point. # event zeek_init() { print "rot13 script is loaded"; } package-manager-3.0.1/testing/baselines/tests.user-mode/0000755000175000017500000000000014565163601021316 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.user-mode/output0000644000175000017500000000172614565163601022607 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. [sources] one = <...>/sources/one [paths] state_dir = <...>/zeekroot/var/lib/zkg script_dir = <...>/zeekroot/share/zeek/site plugin_dir = <...>/zeekroot/lib/zeek/plugins bin_dir = <...>/zeekroot/var/lib/zkg/bin zeek_dist = [templates] default = https://github.com/zeek/package-template Installing "one/alice/foo" Installed "one/alice/foo" (main) Loaded "one/alice/foo" one/alice/foo (installed: main) Installing "zeek/alice/bar" Installed "zeek/alice/bar" (master) Loaded "zeek/alice/bar" zeek/alice/bar (installed: master) Successfully wrote config file to <...>/home/testuser/.zkg/config [sources] [paths] state_dir = <...>/home/testuser/.zkg script_dir = <...>/home/testuser/.zkg/script_dir plugin_dir = <...>/home/testuser/.zkg/plugin_dir bin_dir = <...>/home/testuser/.zkg/bin zeek_dist = [templates] default = https://github.com/zeek/package-template package-manager-3.0.1/testing/baselines/tests.user_vars/0000755000175000017500000000000014565163601021427 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.user_vars/state.logs.foo-build.log10000644000175000017500000000026314565163601026156 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. === STDERR === === STDOUT === /home/jon/sandbox /usr/local /usr package-manager-3.0.1/testing/baselines/tests.user_vars/state.logs.foo-build.log20000644000175000017500000000031314565163601026153 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. === STDERR === === STDOUT === /home/jon/sandbox2 /usr/local Initial description is here package-manager-3.0.1/testing/baselines/tests.user_vars/state.testing.foo.clones.test.log0000644000175000017500000000021714565163601027750 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. Initial description is here package-manager-3.0.1/testing/baselines/tests.plugin_tarfile/0000755000175000017500000000000014565163601022422 5ustar rharhapackage-manager-3.0.1/testing/baselines/tests.plugin_tarfile/scripts.packages.rot13.__load__.zeek0000644000175000017500000000045514565163601031233 0ustar rharha### BTest baseline data generated by btest-diff. Do not edit. Use "btest -U/-u" to update. Requires BTest >= 0.63. # # This is loaded when a user activates the plugin. Include scripts here that should be # loaded automatically at that point. # event zeek_init() { print "rot13 script is loaded"; } package-manager-3.0.1/testing/Makefile0000644000175000017500000000061014565163601015745 0ustar rharha.PHONY: all all: btest-installation-check @btest -j -f diag.log .PHONY: btest-installation-check btest-installation-check: @type btest > /dev/null 2>&1 || \ { \ echo "btest was not located in PATH."; \ echo "Install it and/or make sure it is in PATH."; \ echo "E.g. you could use the following command to install it:"; \ echo "\tpip3 install btest"; \ echo ; \ exit 1; \ } package-manager-3.0.1/testing/btest.cfg0000644000175000017500000000066414565163601016120 0ustar rharha[btest] TestDirs = tests TmpDir = %(testbase)s/.tmp IgnoreDirs = .svn CVS .tmp .git IgnoreFiles = *.tmp *.swp .DS_Store .gitignore BaselineDir = baselines Initializer = %(testbase)s/scripts/initializer MinVersion = 0.63 [environment] ORIGPATH=%(default_path)s PATH=%(testbase)s/scripts:%(default_path)s SCRIPTS=%(testbase)s/scripts SOURCES=%(testbase)s/sources PACKAGES=%(testbase)s/packages TEMPLATES=%(testbase)s/templates package-manager-3.0.1/testing/tests/0000755000175000017500000000000014565163601015452 5ustar rharhapackage-manager-3.0.1/testing/tests/builtin-spicy-version0000644000175000017500000000106514565163601021655 0ustar rharha# @TEST-DOC: Ensure a package that requires a certain spicy-plugin version works with built-in spicy. # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo > output # @TEST-EXEC: btest-diff output # @TEST-EXEC: btest-diff package.log # export LOG=$(pwd)/package.log ( cd packages/foo cat >>zkg.meta <> $LOG build_command = echo "Building foo" >> $LOG depends = spicy-plugin >=6.0.0 EOF git commit -am 'foo: depends on spicy-plugin' ) package-manager-3.0.1/testing/tests/remove0000644000175000017500000000034214565163601016671 0ustar rharha# @TEST-EXEC: zkg install foo bar baz # @TEST-EXEC: test -f scripts/packages/foo/__load__.zeek # @TEST-EXEC: zkg remove foo # @TEST-EXEC: test ! -d scripts/packages/foo # @TEST-EXEC: btest-diff scripts/packages/packages.zeek package-manager-3.0.1/testing/tests/plugin_tarfile0000644000175000017500000000074514565163601020407 0ustar rharha# @TEST-REQUIRES: type zeek-config # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install rot13 # @TEST-EXEC: test -f plugins/packages/rot13/__bro_plugin__ || test -f plugins/packages/rot13/__zeek_plugin__ # @TEST-EXEC: btest-diff scripts/packages/rot13/__load__.zeek cd packages/rot13 echo "plugin_dir = build/Demo_Rot13.tgz" >> zkg.meta git commit -am 'new stuff' cd ../.. echo "$(pwd)/packages/rot13" >> sources/one/bob/zkg.index cd sources/one git commit -am 'add rot13 package' package-manager-3.0.1/testing/tests/install-on-tty0000644000175000017500000000155314565163601020277 0ustar rharha# This test installs a package with a "slow" pseudo build command, once using a # pretend TTY, and once normally. It then verifies that we see progress dots # only in the TTY. This requires the "script" command for TTY fakery. # @TEST-REQUIRES: script --version # @TEST-EXEC: bash %INPUT # https://stackoverflow.com/questions/32910661/pretend-to-be-a-tty-in-bash-for-any-command faketty () { script -qefc "$(printf "%q " "$@")" /dev/null } # Add a build command to the package that takes at least as long as it takes zkg # to produce progress dots. ( cd $(pwd)/packages/foo echo 'build_command = sleep 2' >>zkg.meta git add zkg.meta git commit -am 'build slowly' ) faketty zkg install foo >output.tty zkg uninstall --force foo zkg install foo >output.notty grep 'Installing' output.tty | grep -q '\.' grep 'Installing' output.notty | grep -v -q '\.' package-manager-3.0.1/testing/tests/upgrade-test-fail0000644000175000017500000000207014565163601020711 0ustar rharha# @TEST-DOC: Upgrading fails when a test_comamnd fails, even with --force. # @TEST-REQUIRES: type zeek-config # @TEST-EXEC: (cd packages/foo && echo 'print "foo 1.0.2";' >> __load__.zeek && git tag -a 1.0.2 -m 1.0.2 ) # @TEST-EXEC: zkg install foo >>out 2>&1 # @TEST-EXEC: bash improve-foo-tag-1.0.3 # @TEST-EXEC: zkg refresh # # @TEST-EXEC: echo "=== upgrade without skiptests" >>out # @TEST-EXEC-FAIL: zkg upgrade --force >>out 2>&1 # The upgrade failed, no 1.0.3 to be found in the __load__.zeek file # @TEST-EXEC-FAIL: grep -F '1.0.3' scripts/packages/foo/__load__.zeek >>out # # @TEST-EXEC: echo "=== upgrade with skiptest" >>out # @TEST-EXEC: zkg upgrade --force --skiptests >>out 2>&1 # @TEST-EXEC: grep -F '1.0.3' scripts/packages/foo/__load__.zeek >>out # # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff out @TEST-START-FILE improve-foo-tag-1.0.3 set -e cd packages/foo echo 'test_command = exit 1' >> zkg.meta echo 'print "foo 1.0.3";' > __load__.zeek git add zkg.meta git commit -am 'add a test' git tag -a 1.0.3 -m 1.0.3 @TEST-END-FILE package-manager-3.0.1/testing/tests/unsatisfied-version-tag-dependency0000644000175000017500000000031614565163601024263 0ustar rharha # @TEST-EXEC: bash %INPUT # @TEST-EXEC-FAIL: zkg install foo >fail.out 2>&1 # @TEST-EXEC: btest-diff fail.out cd packages/foo echo 'depends = bar =1.0.0' >> zkg.meta git commit -am 'depend on bar 1.0.0' package-manager-3.0.1/testing/tests/test0000644000175000017500000000143114565163601016353 0ustar rharha# "zkg test" internally requires zeek-config # @TEST-REQUIRES: type zeek-config # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg test rot13 # @TEST-EXEC: bash we_need_to_go_deeper # @TEST-EXEC-FAIL: zkg test rot13 echo "$(pwd)/packages/rot13" >> sources/one/bob/zkg.index cd sources/one git commit -am 'add rot13 package' cd ../../packages/rot13 echo 'depends = bar *' >> zkg.meta echo -e "@load bar\n$(cat scripts/Demo/Rot13/__load__.zeek)" > scripts/Demo/Rot13/__load__.zeek cd testing/Baseline/tests.main echo "rot13 plugin is loaded" > output echo "bar loaded" >> output echo "rot13 script is loaded" >> output git commit -am 'new stuff' @TEST-START-FILE we_need_to_go_deeper cd packages/rot13 echo 'hello' > testing/Baseline/tests.rot13/output git commit -am 'new stuff' @TEST-END-FILE package-manager-3.0.1/testing/tests/info0000644000175000017500000000242014565163601016326 0ustar rharha# @TEST-EXEC: zkg install foo # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg install bar # @TEST-EXEC: zkg info foo | grep -v 'url:' | grep -v 'current_hash' > foo.info # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff foo.info # @TEST-EXEC: zkg info bar | grep -v 'url:' | grep -v 'current_hash' > bar.info # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff bar.info # @TEST-EXEC: zkg remove bar # @TEST-EXEC: zkg info installed | grep -v 'url:' | grep -v 'current_hash' > installed.info # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff installed.info cd packages/foo echo 'print "foo 1.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 cd ../bar echo 'print "bar+ loaded";' > __load__.zeek git commit -am 'new stuff' echo 'print "bar 1.0.0 loaded";' > __load__.zeek echo 'tags = sega sunset' >> zkg.meta echo 'description = bar bar bar' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar++ loaded";' > __load__.zeek git commit -am 'new stuff' package-manager-3.0.1/testing/tests/dependency-ordering0000644000175000017500000000504714565163601021330 0ustar rharha# This test puts in place a dependecy tree, installs the tree's root package, # and verifies that package builds and testing honors the dependency ordering: # depended-upon packages need to be built and available prior to dependers. # The package tree: # foo ---> bar ----> grault # \ / # baz -> corge # # To verify the ordering we use package executables that dependers require both # in test_command and build_command. This leverages the fact that executables # transparently appear in a bin/ directory in the test staging area. foo's build # & test call executables that bar and baz install; baz's build & test call # corge's executable, bar and corge call grault's executable. # We trigger package tests in the below, for which zkg internally looks # for zeek-config. So require a Zeek install: # @TEST-REQUIRES: type zeek-config # @TEST-EXEC: BUILDLOG=$(pwd)/build.log bash %INPUT # After testing, the packages get installed up the dependency chain, installing # their executables into the bin folder at the test's toplevel. That directory # isn't automatically in the path, so add it. # @TEST-EXEC: PATH=$(pwd)/bin:$PATH zkg install foo # Verify the order and number of times the packages got built. # @TEST-EXEC: btest-diff build.log add_executable() { echo "echo from $1" >$1 chmod +x $1 git add $1 } ( cd packages/foo cat >>zkg.meta <>$BUILDLOG test_command = bar && baz depends = bar * baz * EOF git commit -am 'foo: depend on bar and baz, add commands' ) ( cd packages/bar cat >>zkg.meta <>$BUILDLOG test_command = grault executables = bar depends = grault * EOF add_executable bar git commit -am 'bar: depend on grault, add executable and commands' ) ( cd packages/baz cat >>zkg.meta <>$BUILDLOG test_command = corge executables = baz depends = corge * EOF add_executable baz git commit -am 'baz: depend on corge, add executable and commands' ) ( cd packages/corge cat >>zkg.meta <>$BUILDLOG test_command = grault executables = corge depends = grault * EOF add_executable corge git commit -am 'corge: depend on grault, add executable and commands' ) ( cd packages/grault cat >>zkg.meta <>$BUILDLOG executables = grault EOF add_executable grault git commit -am 'grault: add executable and build command' ) package-manager-3.0.1/testing/tests/builtin-info0000644000175000017500000000035714565163601020001 0ustar rharha# @TEST-DOC: Using info on a built-in package works # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC: zkg info spicy-plugin | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+/X.X.X/g' >out # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/metadata-depends0000644000175000017500000000271614565163601020603 0ustar rharha# @TEST-EXEC: zkg install foo # @TEST-EXEC: zkg list installed > no_depends.out # @TEST-EXEC: btest-diff no_depends.out # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg upgrade foo # @TEST-EXEC: zkg list installed > upgrade_depends.out # @TEST-EXEC: btest-diff upgrade_depends.out # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg install foo # @TEST-EXEC: zkg list installed > install_depends.out # @TEST-EXEC: btest-diff install_depends.out # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg bundle test.bundle --manifest foo # @TEST-EXEC: zkg unbundle test.bundle # @TEST-EXEC: zkg list installed > bundle_depends.out # @TEST-EXEC: btest-diff bundle_depends.out cd packages/foo echo 'depends = bar *' >> zkg.meta git commit -am 'new stuff' cd ../bar echo 'depends = baz >=1.0.0' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 default_branch_name=$( cd ../grault && git rev-parse --abbrev-ref HEAD ) cd ../baz echo "depends = grault branch=${default_branch_name}" >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "2.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 cd ../grault echo 'depends = corge ==1.0.1' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 cd ../corge git tag -a 1.0.0 -m 1.0.0 echo 'depends = foo * bar *' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "2.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 package-manager-3.0.1/testing/tests/builtin-install0000644000175000017500000000036614565163601020514 0ustar rharha# @TEST-DOC: Ensure that installing spicy-plugin fails if it's built-in (default for 6.0) # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC-FAIL: zkg install spicy-plugin >out 2>&1 # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/template-create0000644000175000017500000000366714565163601020465 0ustar rharha# This test verifies template instantiation. # Provide variable via --user-var and ensure resulting package installs # @TEST-EXEC: zkg create --packagedir out1 --template $TEMPLATES/foo --user-var name=test1 >output # @TEST-EXEC: zkg install ./out1 # Provide variable via environment and ensure resulting package installs. # @TEST-EXEC: name=test2 zkg create --packagedir out2 --template $TEMPLATES/foo >>output # @TEST-EXEC: zkg install ./out2 # # Verify the README does not exist -- it's provided by a template feature # @TEST-EXEC: test ! -f out2/README # Same as first test, but now request the readme feature. This uses # --force to suppress user input, making user var resolution fall back # to the default for the readme parameter. # @TEST-EXEC: zkg create --force --packagedir out3 --template $TEMPLATES/foo --feature readme --user-var name=test3 >>output # @TEST-EXEC: zkg install ./out3 # @TEST-EXEC: btest-diff out3/README # @TEST-EXEC: btest-diff out3/scripts/main.zeek # # Verify the zkg.meta content. # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-zkg-version btest-diff out3/zkg.meta # Create a package from a git-versioned template repo -- this has different # information in zkg.meta: # @TEST-EXEC: zkg create --packagedir out4 --template templates/foo --user-var name=test4 >>output # @TEST-EXEC: TEST_DIFF_CANONIFIER="$SCRIPTS/diff-remove-zkg-version | $SCRIPTS/diff-remove-zkg-meta-commit" btest-diff out4/zkg.meta # zkg should now have the first three packages installed. # @TEST-EXEC: zkg list >>output # Fail to provide a user variable when using --force (thus suppressing input). This should fail. # @TEST-EXEC-FAIL: zkg create --force --packagedir out4 --template $TEMPLATES/foo >>output 2>&1 # Request an unknown feature. This should fail. # @TEST-EXEC-FAIL: zkg create --packagedir out5 --template $TEMPLATES/foo --feature doesntexist >>output 2>&1 # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff output package-manager-3.0.1/testing/tests/builtin-remove0000644000175000017500000000036314565163601020340 0ustar rharha# @TEST-DOC: Ensure that removing spicy-plugin fails if it's built-in (default for 6.0) # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC-FAIL: zkg remove spicy-plugin >out 2>&1 # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/install0000644000175000017500000000445414565163601017052 0ustar rharha# @TEST-EXEC: zkg install foo # @TEST-EXEC: btest-diff scripts/packages/foo/__load__.zeek # @TEST-EXEC: zkg install alice/bar # @TEST-EXEC: btest-diff scripts/packages/bar/__load__.zeek # @TEST-EXEC: zkg install one/alice/baz # @TEST-EXEC: btest-diff scripts/packages/baz/__load__.zeek # @TEST-EXEC: zkg install $(pwd)/packages/qux # @TEST-EXEC: btest-diff scripts/packages/qux/__load__.zeek # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install corge # @TEST-EXEC: btest-diff scripts/packages/corge/__load__.zeek # @TEST-EXEC: zkg remove corge # @TEST-EXEC: ( cd $(pwd)/packages/corge && git rev-parse --abbrev-ref HEAD ) > default_branch_name # @TEST-EXEC: zkg install corge --version=$(cat default_branch_name) # @TEST-EXEC: cp scripts/packages/corge/__load__.zeek corge.master # @TEST-EXEC: btest-diff corge.master cd packages/corge git tag -a 1.0.0 -m 1.0.0 echo 'print "hello";' >> __load__.zeek git commit -am 'new stuff' cd ../grault echo 'print "grault 1.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "grault 1.0.1";' > __load__.zeek git commit -am 'new stuff' git tag -a v1.0.1 -m 1.0.1 echo 'print "grault 1.0.2";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 # @TEST-EXEC: zkg install grault # @TEST-EXEC: btest-diff scripts/packages/grault/__load__.zeek # @TEST-EXEC: zkg remove grault # @TEST-EXEC: zkg install grault --version=v1.0.1 # @TEST-EXEC: cp scripts/packages/grault/__load__.zeek grault.1.0.1 # @TEST-EXEC: btest-diff grault.1.0.1 # # @TEST-EXEC: zkg remove grault # @TEST-EXEC: (cd packages/grault && git checkout --detach) # @TEST-EXEC: zkg install $(pwd)/packages/grault # @TEST-EXEC: cp scripts/packages/grault/__load__.zeek grault.detached # @TEST-EXEC: btest-diff grault.detached # @TEST-EXEC: zkg info grault | grep tracking_method > grault.info # @TEST-EXEC: btest-diff grault.info # # @TEST-EXEC: zkg remove grault # @TEST-EXEC: (cd packages/grault && git checkout 1.0.0) # @TEST-EXEC: (cd packages/grault && git rev-parse HEAD) > grault.head.hash # @TEST-EXEC: zkg install grault --version=$(cat grault.head.hash) # @TEST-EXEC: cp scripts/packages/grault/__load__.zeek grault.detached.head # @TEST-EXEC: btest-diff grault.detached.head # @TEST-EXEC: zkg info grault | grep tracking_method > grault.info # @TEST-EXEC: btest-diff grault.info package-manager-3.0.1/testing/tests/package_base0000644000175000017500000000105314565163601017761 0ustar rharha# This test involves package testing, for which zkg internally requires # zeek-config. So require a Zeek install: # @TEST-REQUIRES: type zeek-config # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo # @TEST-EXEC: btest-diff state/logs/foo-build.log # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-cwd btest-diff state/testing/foo/clones/test.log cd packages/foo echo 'build_command = cd "%(package_base)s" && ls' >> zkg.meta echo 'test_command = echo "%(package_base)s" > %(package_base)s/test.log' >> zkg.meta git commit -am 'new stuff' package-manager-3.0.1/testing/tests/bindir-relocate0000644000175000017500000000127714565163601020447 0ustar rharha# @TEST-EXEC: bash %INPUT # # @TEST-EXEC: zkg install foo # @TEST-EXEC: test -L bin/exec1 && test -L bin/exec2 # # @TEST-EXEC: sed -i.bak 's/\(^bin_dir.*\)/\1-new/g' config # @TEST-EXEC: zkg list # @TEST-EXEC: test -L bin-new/exec1 && test -L bin-new/exec2 # @TEST-EXEC: ./bin-new/exec1 >>output # @TEST-EXEC: ./bin-new/exec2 >>output # @TEST-EXEC: test '!' -e bin/exec1 && test '!' -e bin/exec2 # @TEST-EXEC: test '!' -d bin # # @TEST-EXEC: btest-diff output cd packages/foo echo "test_command = true" >>zkg.meta echo "executables = x/exec1 x/exec2 " >>zkg.meta mkdir x echo "echo from exec1" >x/exec1 echo "echo from exec2" >x/exec2 chmod +x x/exec1 x/exec2 git add * git commit -m 'new stuff' package-manager-3.0.1/testing/tests/builtin-unpin0000644000175000017500000000036314565163601020174 0ustar rharha# @TEST-DOC: Ensure that unpinning spicy-plugin fails if it's built-in (default for 6.0) # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC-FAIL: zkg unpin spicy-plugin >out 2>&1 # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/install-invalid0000644000175000017500000000144714565163601020475 0ustar rharha# Test invalid package names and paths # @TEST-EXEC-FAIL: bash %INPUT 2>output # @TEST-EXEC: btest-diff output CONFIG=$(pwd)/config mkdir -p invalid cp -R ./packages/bar ./invalid/\ fronting-whitespace (cd ./invalid/\ fronting-whitespace && zkg --config=$CONFIG install --force .) cp -R ./packages/bar './invalid/trailing-whitespace ' (cd './invalid/trailing-whitespace ' && zkg --config=$CONFIG install --force . ) zkg --config=$CONFIG install --force ./packages/doesntexist mkdir ./packages/notagitrepo zkg --config=$CONFIG install --force ./packages/notagitrepo mkdir ./packages/dirtyrepo ( cd ./packages/dirtyrepo && git init . && touch README && git add README && git commit -m "Initial commit") echo README > ./packages/dirtyrepo/README zkg --config=$CONFIG install --force ./packages/dirtyrepo package-manager-3.0.1/testing/tests/install-bro-pkg-warning0000644000175000017500000000033014565163601022041 0ustar rharha# @TEST-EXEC: (cd packages/foo; git mv zkg.meta bro-pkg.meta; git commit -m 'Use bro-pkg.meta') # @TEST-EXEC: zkg install foo 2> stderr # @TEST-EXEC: TEST_DIFF_CANONIFIER='sed -r "s/[0-9]{2}/XX/g"' btest-diff stderr package-manager-3.0.1/testing/tests/builtin-spicy-version-error0000644000175000017500000000120714565163601023002 0ustar rharha# @TEST-DOC: Package depends for built-in Spicy cannot be fulfilled with built-in package. # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC: bash %INPUT # @TEST-EXEC-FAIL: zkg install foo >output.orig 2>&1 # @TEST-EXEC: sed -r 's/[0-9]+\.[0-9]+\.[0-9]+/X.X.X/g' output # @TEST-EXEC: btest-diff output # @TEST-EXEC: test ! -f package.log # export LOG=$(pwd)/package.log ( cd packages/foo cat >>zkg.meta <> $LOG build_command = echo "Building foo" >> $LOG depends = spicy-plugin <6.0.0 EOF git commit -am 'foo: depends on old spicy-plugin' ) package-manager-3.0.1/testing/tests/builtin-pin0000644000175000017500000000035714565163601017634 0ustar rharha# @TEST-DOC: Ensure that pinning spicy-plugin fails if it's built-in (default for 6.0) # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC-FAIL: zkg pin spicy-plugin >out 2>&1 # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/bundle-unsatisfied-dependency0000644000175000017500000000203214565163601023273 0ustar rharha# @TEST-DOC: Create a bundle by hand with a package depending on "spicy-plugin<6.0.0" which isn't satisfiable. Observe a warning during unbundling. # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # @TEST-REQUIRES: tar --version # # @TEST-EXEC: bash %INPUT # # @TEST-EXEC: zkg unbundle bundle.tar >> output # @TEST-EXEC: zkg list >> output # @TEST-EXEC: btest-diff output # @TEST-EXEC: TEST_DIFF_CANONIFIER='sed -r "s/[0-9]{2}/XX/g" | sed -r "s/\(.\..\..\)/(X.X.X)/g" | $SCRIPTS/diff-remove-abspath btest-diff' btest-diff .stderr # @TEST-EXEC: btest-diff package.log # export LOG=$(pwd)/package.log ( cd packages/foo cat >>zkg.meta <> $LOG build_command = echo "Building foo" >> $LOG depends = spicy-plugin <6.0.0 EOF git commit -am 'foo: depends on spicy-plugin' git checkout -b origin/main ) # Create a bundle mkdir the-bundle cp -R packages/foo ./the-bundle echo -e '[bundle]\n/one/alice/foo = main' >> the-bundle/manifest.txt tar -cf bundle.tar -C ./the-bundle . package-manager-3.0.1/testing/tests/metadata-suggests0000644000175000017500000000273314565163601021024 0ustar rharha# @TEST-EXEC: zkg install foo # @TEST-EXEC: zkg list installed > no_suggests.out # @TEST-EXEC: btest-diff no_suggests.out # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg upgrade foo # @TEST-EXEC: zkg list installed > upgrade_suggests.out # @TEST-EXEC: btest-diff upgrade_suggests.out # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg install foo # @TEST-EXEC: zkg list installed > install_suggests.out # @TEST-EXEC: btest-diff install_suggests.out # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg bundle test.bundle --manifest foo # @TEST-EXEC: zkg unbundle test.bundle # @TEST-EXEC: zkg list installed > bundle_suggests.out # @TEST-EXEC: btest-diff bundle_suggests.out cd packages/foo echo 'suggests = bar *' >> zkg.meta git commit -am 'new stuff' cd ../bar echo 'suggests = baz >=1.0.0' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 default_branch_name=$( cd ../grault && git rev-parse --abbrev-ref HEAD ) cd ../baz echo "suggests = grault branch=${default_branch_name}" >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "2.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 cd ../grault echo 'suggests = corge ==1.0.1' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 cd ../corge git tag -a 1.0.0 -m 1.0.0 echo 'suggests = foo * bar *' >> zkg.meta git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "2.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 package-manager-3.0.1/testing/tests/install-reserved0000644000175000017500000000042714565163601020663 0ustar rharha# Rename a package to live in a directory called "packages" # @TEST-EXEC-FAIL: bash %INPUT 2>output # @TEST-EXEC: btest-diff output CONFIG=$(pwd)/config mkdir -p reserved cp -R ./packages/bar ./reserved/packages cd ./reserved/packages && zkg --config=$CONFIG install --force . package-manager-3.0.1/testing/tests/bindir-install0000644000175000017500000000125414565163601020312 0ustar rharha# "zkg test" internally requires zeek-config # @TEST-REQUIRES: type zeek-config # @TEST-EXEC: bash %INPUT # # @TEST-EXEC: zkg test foo # # @TEST-EXEC: zkg install foo # @TEST-EXEC: test -d bin # @TEST-EXEC: test -L bin/exec1 && test -L bin/exec2 # @TEST-EXEC: ./bin/exec1 >>output # @TEST-EXEC: ./bin/exec2 >>output # @TEST-EXEC: btest-diff output # # @TEST-EXEC: zkg remove foo # @TEST-EXEC: test '!' -e bin/exec1 && test '!' -e bin/exec2 cd packages/foo echo "test_command = true" >>zkg.meta echo "executables = x/exec1 x/exec2 " >>zkg.meta mkdir x echo "echo from exec1" >x/exec1 echo "echo from exec2" >x/exec2 chmod +x x/exec1 x/exec2 git add * git commit -m 'new stuff' package-manager-3.0.1/testing/tests/load0000644000175000017500000000107714565163601016321 0ustar rharha# @TEST-EXEC: zkg install foo bar # @TEST-EXEC: btest-diff scripts/packages/packages.zeek # @TEST-EXEC: zkg unload foo # @TEST-EXEC: cp scripts/packages/packages.zeek after_unload.out # @TEST-EXEC: btest-diff after_unload.out # @TEST-EXEC: zkg load foo # @TEST-EXEC: cp scripts/packages/packages.zeek after_load.out # @TEST-EXEC: btest-diff after_load.out # @TEST-EXEC: zkg install i-have-no-scripts # @TEST-EXEC: zkg load i-have-no-scripts # @TEST-EXEC: cp scripts/packages/packages.zeek after_loading_no_scripts.out # @TEST-EXEC: btest-diff after_loading_no_scripts.out package-manager-3.0.1/testing/tests/installed-dependency-conflict0000644000175000017500000000126614565163601023274 0ustar rharha # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install --version 1.0.0 foo # @TEST-EXEC: zkg list installed > installed-initial.out # @TEST-EXEC-FAIL: zkg install --version 2.0.0 foo >conflict.out 2>&1 # @TEST-EXEC: zkg list installed > installed-final.out # @TEST-EXEC: btest-diff installed-initial.out # @TEST-EXEC: btest-diff conflict.out # @TEST-EXEC: btest-diff installed-final.out cd packages/foo echo 'depends = bar =1.0.0' >> zkg.meta git commit -am 'depend on bar 1.0.0' git tag -a 1.0.0 -m 1.0.0 git checkout HEAD~1 echo 'depends = bar =2.0.0' >> zkg.meta git commit -am 'depend on bar 2.0.0' git tag -a 2.0.0 -m 2.0.0 cd ../bar git tag -a 1.0.0 -m 1.0.0 git tag -a 2.0.0 -m 2.0.0 package-manager-3.0.1/testing/tests/refresh0000644000175000017500000000462614565163601017043 0ustar rharha# @TEST-EXEC: zkg install foo # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg list outdated > outdated.out # @TEST-EXEC: btest-diff outdated.out # @TEST-EXEC: zkg list all > list.out # @TEST-EXEC: btest-diff list.out # With --fail-on-aggregate problems, the following should fail since # there are metadata problems in the package set. # @TEST-EXEC-FAIL: zkg -vvv refresh --aggregate --fail-on-aggregate-problems >agg.fail.out 2>agg.fail.errout.orig # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff agg.fail.out # @TEST-EXEC: grep -v 'built-in package' < agg.fail.errout.orig > agg.fail.errout # @TEST-EXEC: TEST_DIFF_CANONIFIER='$SCRIPTS/diff-canonifier' btest-diff agg.fail.errout # Remove the aggregated metadata so the next invocation has a clean slate # @TEST-EXEC: rm -f state/clones/source/one/aggregate.meta # This time we trigger only warnings for offending packages, commit, and push. # @TEST-EXEC: zkg -vvv refresh --aggregate --push >agg.out 2>agg.errout.orig # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff agg.out # @TEST-EXEC: grep -v 'built-in package' < agg.errout.orig > agg.errout # @TEST-EXEC: TEST_DIFF_CANONIFIER='$SCRIPTS/diff-canonifier' btest-diff agg.fail.errout # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-canonifier btest-diff agg.errout # @TEST-EXEC: zkg search lucky > search.out # @TEST-EXEC: btest-diff search.out # @TEST-EXEC: zkg list all > list_after_agg.out # @TEST-EXEC: btest-diff list_after_agg.out # note: foo's description isn't in the list output since the installed # version's metadata has no 'description' field cd packages/foo echo 'tags = esoteric lucky land' >> zkg.meta echo 'description = This is the foo package description' >> zkg.meta git commit -am 'new stuff' cd .. mkdir new_pkg cd new_pkg git init echo '[package]' > zkg.meta echo 'tags = esoteric lucky land' >> zkg.meta echo 'description = This is the new_pkg package description' >> zkg.meta echo 'print "hello";' >> __load__.zeek git add * git commit -m 'init' cd .. mkdir bad_pkg cd bad_pkg git init echo '[package]' > bad.meta git add * git commit -m 'init' cd ../.. echo "$(pwd)/packages/new_pkg" >> sources/one/alice/zkg.index echo "$(pwd)/packages/bad_pkg" >> sources/one/alice/zkg.index cd sources ( cd one && git commit -am 'add packages' ) # Make it a bare repo so we can push to it mv one one.tmp git clone --bare one.tmp one rm -rf one.tmp package-manager-3.0.1/testing/tests/uninstall0000644000175000017500000000035514565163601017411 0ustar rharha# @TEST-EXEC: zkg install foo bar baz # @TEST-EXEC: test -f scripts/packages/foo/__load__.zeek # @TEST-EXEC: zkg uninstall --force foo # @TEST-EXEC: test ! -d scripts/packages/foo # @TEST-EXEC: btest-diff scripts/packages/packages.zeek package-manager-3.0.1/testing/tests/metadata-script_dir0000644000175000017500000000036014565163601021314 0ustar rharha# @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo # @TEST-EXEC: btest-diff scripts/foo/__load__.zeek cd packages/foo mkdir scripts echo 'script_dir = scripts' >> zkg.meta mv __load__.zeek scripts/ git add * git commit -m 'new stuff' package-manager-3.0.1/testing/tests/list0000644000175000017500000000161414565163601016352 0ustar rharha# @TEST-EXEC: zkg list all > all.out # @TEST-EXEC: btest-diff all.out # @TEST-EXEC: zkg install foo bar # @TEST-EXEC: zkg unload bar # @TEST-EXEC: zkg list installed > installed.out # @TEST-EXEC: btest-diff installed.out # @TEST-EXEC: zkg list > list.out # @TEST-EXEC: cmp list.out installed.out # @TEST-EXEC: zkg list loaded > loaded.out # @TEST-EXEC: btest-diff loaded.out # @TEST-EXEC: zkg list unloaded > unloaded.out # @TEST-EXEC: btest-diff unloaded.out # @TEST-EXEC: zkg list not_installed > not_installed.out # @TEST-EXEC: btest-diff not_installed.out # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg list outdated > outdated.out # @TEST-EXEC: btest-diff outdated.out # @TEST-EXEC: zkg upgrade # @TEST-EXEC: zkg list outdated > after_upgrade.out # @TEST-EXEC: btest-diff after_upgrade.out cd packages/foo echo 'print "hello";' >> __load__.zeek git commit -am 'new stuff' package-manager-3.0.1/testing/tests/user-mode0000644000175000017500000000271714565163601017304 0ustar rharha# Establish a skeletal Zeek installation layout and alternative home directory # @TEST-EXEC: setup-zeek-and-home # Pretending to be the Zeek-bundled zkg, install a package and show it # @TEST-EXEC: zkg-zeek config >>output # @TEST-EXEC: zkg-zeek install --force foo >>output # @TEST-EXEC: zkg-zeek list >>output # At this point, zkg should have crated manifest and package state in # the Zeek install tree. The home directory should still be left # alone. # @TEST-EXEC: test -f zeekroot/var/lib/zkg/manifest.json # @TEST-EXEC: test -f zeekroot/share/zeek/site/packages/foo/__load__.zeek # @TEST-EXEC: test ! -d home/testuser/.zkg # Switching to user mode, install a different package. # @TEST-EXEC: zkg-zeek --user --extra-source zeek=$(pwd)/sources/one install --force bar >>output # # Only that package should get listed, and zkg should have built up # state in the home directory. # @TEST-EXEC: zkg-zeek --user list >>output # @TEST-EXEC: test -f home/testuser/.zkg/manifest.json # @TEST-EXEC: test ! -f home/testuser/.zkg/script_dir/packages/foo/__load__.zeek # @TEST-EXEC: test -f home/testuser/.zkg/script_dir/packages/bar/__load__.zeek # So far this didn't need a config file, but we can produce one: # @TEST-EXEC: test ! -f home/testuser/.zkg/config # @TEST-EXEC: zkg-zeek --user autoconfig >>output # @TEST-EXEC: test -f home/testuser/.zkg/config # @TEST-EXEC: zkg-zeek --user config >>output # @TEST-EXEC: TEST_DIFF_CANONIFIER="$SCRIPTS/diff-remove-cwd" btest-diff output package-manager-3.0.1/testing/tests/purge0000644000175000017500000000065114565163601016521 0ustar rharha# @TEST-EXEC: zkg install foo bar baz # @TEST-EXEC: test -f scripts/packages/foo/__load__.zeek # @TEST-EXEC: test -f scripts/packages/bar/__load__.zeek # @TEST-EXEC: test -f scripts/packages/baz/__load__.zeek # @TEST-EXEC: zkg purge # @TEST-EXEC: test ! -d scripts/packages/foo # @TEST-EXEC: test ! -d scripts/packages/bar # @TEST-EXEC: test ! -d scripts/packages/baz # @TEST-EXEC: btest-diff scripts/packages/packages.zeek package-manager-3.0.1/testing/tests/bundled-zkg0000644000175000017500000000123714565163601017606 0ustar rharha# @TEST-DOC: Simulate bundling of zkg in Zeek and ensure zkg works. This got broken during a cleanup previously. # @TEST-EXEC: bash %INPUT > output # @TEST-EXEC: btest-diff output set -eu # Place zkg and zeekpkg into bin and lib/python directories and do # the templating. mkdir -p install-dir/{bin,lib/python} cp -R $TEST_BASE/../zeekpkg ./install-dir/lib/python cp $TEST_BASE/../zkg ./install-dir/bin sed -i "s,@PY_MOD_INSTALL_DIR@,$(pwd)/install-dir/lib/python," ./install-dir/bin/zkg export PATH=$(pwd)/install-dir/bin:$PATH # Ensure we're using the right zkg command -v zkg | sed -E 's,^.*/(.*/.*/.*/zkg),/\1,' zkg --version | sed 's/zkg [0-9.-]*/zkg xxx/' package-manager-3.0.1/testing/tests/builtin-test0000644000175000017500000000036014565163601020017 0ustar rharha# @TEST-DOC: Ensure that testing spicy-plugin fails if it's built-in (default for 6.0) # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC-FAIL: zkg test spicy-plugin >out 2>&1 # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/user_vars0000644000175000017500000000213414565163601017406 0ustar rharha# This test involves package testing, for which zkg internally requires # zeek-config. So require a Zeek install: # @TEST-REQUIRES: type zeek-config # @TEST-EXEC: bash %INPUT # @TEST-EXEC: LAST_VAR=/home/jon/sandbox zkg install foo # @TEST-EXEC: cp state/logs/foo-build.log state/logs/foo-build.log1 # @TEST-EXEC: btest-diff state/logs/foo-build.log1 # @TEST-EXEC: zkg install --user-var TEST_VAR="Initial description is here" --user-var LAST_VAR=/home/jon/sandbox2 foo # @TEST-EXEC: cp state/logs/foo-build.log state/logs/foo-build.log2 # @TEST-EXEC: btest-diff state/logs/foo-build.log2 # @TEST-EXEC: btest-diff state/testing/foo/clones/test.log cd packages/foo echo 'user_vars =' >> zkg.meta echo ' TEST_VAR [/usr] "First description is here"' >> zkg.meta echo ' ANOTHER_VAR [/usr/local] "Second description is here"' >> zkg.meta echo ' LAST_VAR [/opt] "Last description is here"' >> zkg.meta echo 'build_command = echo "%(LAST_VAR)s" && echo "%(ANOTHER_VAR)s" && echo "%(TEST_VAR)s"' >> zkg.meta echo 'test_command = echo "%(TEST_VAR)s" > %(package_base)s/test.log' >> zkg.meta git commit -am 'new stuff' package-manager-3.0.1/testing/tests/install-force-test-fail0000644000175000017500000000152214565163601022025 0ustar rharha# @TEST-DOC: Test that a failing test_command with --force causes zkg install to fail and the package is not loaded. Also, verify that --skiptests can override that. # @TEST-REQUIRES: type zeek-config # # @TEST-EXEC: bash %INPUT # @TEST-EXEC: echo "=== install without skiptests" >>out # @TEST-EXEC-FAIL: zkg install --force foo >>out 2>&1 # @TEST-EXEC: echo "have_load_foo=$(grep -c '@load.*foo' ./scripts/packages/packages.zeek)" >>out # # @TEST-EXEC: rm -rf ./state # @TEST-EXEC: echo "=== install with skiptests" >>out # @TEST-EXEC: zkg install --force --skiptests foo >>out 2>&1 # @TEST-EXEC: echo "have_load_foo=$(grep -c '@load.*foo' ./scripts/packages/packages.zeek)" >>out # # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff out cd packages/foo echo 'test_command = exit 1' >> zkg.meta git commit -am 'add a test' package-manager-3.0.1/testing/tests/branch-based-dependency0000644000175000017500000000053314565163601022023 0ustar rharha # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install --version release/1.0 foo # @TEST-EXEC: zkg list installed > installed.out # @TEST-EXEC: btest-diff installed.out cd packages/foo git checkout -b release/1.0 echo 'depends = bar branch=release/1.0' >> zkg.meta git commit -am 'depend on bar release/1.0' cd ../bar git checkout -b release/1.0 package-manager-3.0.1/testing/tests/load-zeek-dependency0000644000175000017500000000102014565163601021355 0ustar rharha# @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo # @TEST-EXEC: zkg unload foo # @TEST-EXEC: zkg list loaded >nothing-loaded.out # @TEST-EXEC: btest-diff nothing-loaded.out # @TEST-EXEC: zkg load foo # @TEST-EXEC: zkg list loaded >foo-loaded.out # @TEST-EXEC: btest-diff foo-loaded.out # The logic for dependency-aware (un)loading should just silently ignore "zeek" # as a special dependency and not a real package to consider (un)loading. cd packages/foo echo 'depends = zeek *' >> zkg.meta git commit -am 'new stuff' package-manager-3.0.1/testing/tests/plugin0000644000175000017500000000212014565163601016666 0ustar rharha# @TEST-REQUIRES: type zeek-config # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install rot13 # @TEST-EXEC: test -f plugins/packages/rot13/__bro_plugin__ || test -f plugins/packages/rot13/__zeek_plugin__ # @TEST-EXEC: btest-diff scripts/packages/rot13/__load__.zeek # Unloading the package should also disable the plugin, which we # detect via the renamed __bro_plugin__ magic file. # @TEST-EXEC: zkg unload rot13 # @TEST-EXEC: test ! -f plugins/packages/rot13/__bro_plugin__ && test ! -f plugins/packages/rot13/__zeek_plugin__ # @TEST-EXEC: test -f plugins/packages/rot13/__bro_plugin__.disabled || test -f plugins/packages/rot13/__zeek_plugin__.disabled # (Re-)loading the package should also (re-)enable the plugin. # @TEST-EXEC: zkg load rot13 # @TEST-EXEC: test -f plugins/packages/rot13/__bro_plugin__ || test -f plugins/packages/rot13/__zeek_plugin__ # @TEST-EXEC: test ! -f plugins/packages/rot13/__bro_plugin__.disabled && test ! -f plugins/packages/rot13/__zeek_plugin__.disabled echo "$(pwd)/packages/rot13" >> sources/one/bob/zkg.index cd sources/one git commit -am 'add rot13 package' package-manager-3.0.1/testing/tests/install-force-build-fail0000644000175000017500000000115614565163601022150 0ustar rharha# @TEST-DOC: Test that a failing build_command fails the package installation, no matter what. # @TEST-REQUIRES: type zeek-config # # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install bar # # @TEST-EXEC-FAIL: zkg install --force --skiptests foo >>out 2>&1 # @TEST-EXEC: echo "have_load_bar=$(grep -c '@load.*bar' ./scripts/packages/packages.zeek)" >>out # @TEST-EXEC: echo "have_load_foo=$(grep -c '@load.*foo' ./scripts/packages/packages.zeek)" >>out # # @TEST-EXEC: TEST_DIFF_CANONIFIER=$SCRIPTS/diff-remove-abspath btest-diff out cd packages/foo echo 'build_command = exit 1' >> zkg.meta git commit -am 'build fails' package-manager-3.0.1/testing/tests/builtin-unload0000644000175000017500000000036414565163601020326 0ustar rharha# @TEST-DOC: Ensure that unloading spicy-plugin fails if it's built-in (default for 6.0) # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC-FAIL: zkg unload spicy-plugin >out 2>&1 # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/metadata-config_files0000644000175000017500000000233214565163601021602 0ustar rharha# @TEST-EXEC: bash setup # @TEST-EXEC: zkg install foo bar # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg upgrade foo # @TEST-EXEC: btest-diff scripts/foo/config.zeek # @TEST-EXEC: btest-diff scripts/bar/config.zeek # @TEST-EXEC: cp state/backups/one/alice/foo/config.zeek* foo_backup.zeek # @TEST-EXEC: btest-diff foo_backup.zeek # @TEST-EXEC: zkg install bar --version=2.0.0 # @TEST-EXEC: cp scripts/bar/config.zeek barconfig2.zeek # @TEST-EXEC: btest-diff barconfig2.zeek # @TEST-EXEC: cp state/backups/one/alice/bar/config.zeek* bar_backup.zeek # @TEST-EXEC: btest-diff bar_backup.zeek @TEST-START-FILE setup cd packages/foo echo 'print "foo orig";' > config.zeek echo 'config_files = config.zeek' >> zkg.meta git add * git commit -am 'new stuff' cd ../bar echo 'print "bar 1.0.0";' > config.zeek echo 'config_files = config.zeek' >> zkg.meta git add * git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 @TEST-END-FILE echo 'print "foo mod";' > scripts/foo/config.zeek echo 'print "bar mod";' > scripts/bar/config.zeek cd packages/foo echo 'print "foo new";' > config.zeek git commit -am 'new stuff' cd ../bar echo 'print "bar 2.0.0";' > config.zeek git commit -am 'new stuff' git tag -a 2.0.0 -m 2.0.0 package-manager-3.0.1/testing/tests/dependency-management0000644000175000017500000000252114565163601021625 0ustar rharha# This test puts in place a dependency chain and then verifies zkg's behavior # when the state of individual packages in that chain changes. The chain: # # foo -> bar (*) -> baz (>=1.0.0) -> grault (==1.0.0) -> corge (==1.0.0) # @TEST-EXEC: bash %INPUT # @TEST-EXEC: echo '** Install grault' >out # @TEST-EXEC: zkg install grault >>out # @TEST-EXEC: echo '** Unload grault' >>out # @TEST-EXEC: zkg unload grault >>out # @TEST-EXEC: echo '** Install foo' >>out # @TEST-EXEC: zkg install foo >>out # @TEST-EXEC: echo '** Unload foo' >>out # @TEST-EXEC: zkg unload foo >>out # @TEST-EXEC: echo '** Load foo' >>out # @TEST-EXEC: zkg load foo >>out # @TEST-EXEC: echo '** Unload bar' >>out # @TEST-EXEC: zkg unload bar>>out # @TEST-EXEC: echo '** Remove grault' >>out # @TEST-EXEC: zkg remove grault >>out # @TEST-EXEC: btest-diff out cd packages/foo echo 'depends = bar *' >> zkg.meta git commit -am 'foo now depends on bar' cd ../bar echo 'depends = baz >=1.0.0' >> zkg.meta git commit -am 'bar now depends on baz >= 1.0.0' git tag -a 1.0.0 -m 1.0.0 cd ../baz echo 'depends = grault ==1.0.0' >> zkg.meta git commit -am 'baz now depends on grault 1.0.0' git tag -a 1.0.0 -m 1.0.0 cd ../grault echo 'depends = corge ==1.0.0' >> zkg.meta git commit -am 'grault now depends on corge 1.0.0' git tag -a 1.0.0 -m 1.0.0 cd ../corge git tag -a 1.0.0 -m 1.0.0 package-manager-3.0.1/testing/tests/pin0000644000175000017500000000267614565163601016176 0ustar rharha# @TEST-EXEC: bash setup_foo # @TEST-EXEC: zkg install foo bar # @TEST-EXEC: zkg pin foo bar # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC-FAIL: zkg upgrade # @TEST-EXEC: btest-diff scripts/packages/foo/__load__.zeek # @TEST-EXEC: btest-diff scripts/packages/bar/__load__.zeek # @TEST-EXEC: zkg unpin foo bar # @TEST-EXEC: zkg upgrade # @TEST-EXEC: cp scripts/packages/foo/__load__.zeek foo_after_unpin.out # @TEST-EXEC: cp scripts/packages/bar/__load__.zeek bar_after_unpin.out # @TEST-EXEC: btest-diff foo_after_unpin.out # @TEST-EXEC: btest-diff bar_after_unpin.out @TEST-START-FILE setup_foo cd packages/foo echo 'print "foo 1.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 @TEST-END-FILE cd packages/foo echo 'print "foo 1.0.3";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.3 -m 1.0.3 echo 'print "foo 1.0.4";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.4 -m 1.0.4 echo 'print "foo master";' > __load__.zeek git commit -am 'new stuff' cd ../bar echo 'print "bar+ loaded";' > __load__.zeek git commit -am 'new stuff' echo 'print "bar 1.0.0 loaded";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar++ loaded";' > __load__.zeek git commit -am 'new stuff' package-manager-3.0.1/testing/tests/builtin-list0000644000175000017500000000044214565163601020014 0ustar rharha# @TEST-DOC: Listing packages with --include-builtin shows built-in packages. # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC: zkg list --include-builtin | grep spicy-plugin | sed -E 's/[0-9]+\.[0-9]+\.[0-9]+/X.X.X/g' >out # @TEST-EXEC: btest-diff out package-manager-3.0.1/testing/tests/validate-manifest0000644000175000017500000000116514565163601020775 0ustar rharha# @TEST-DOC: List installed_packages in manifest.json - regression test for adding built-in packages. # @TEST-EXEC: zkg install foo # @TEST-EXEC: zkg install alice/bar # @TEST-EXEC: zkg install one/alice/baz # @TEST-EXEC: python3 read-manifest.py ./state/manifest.json > packages.txt # @TEST-EXEC: btest-diff packages.txt # @TEST-START-FILE read-manifest.py import json, sys with open(sys.argv[1]) as f: manifest = json.load(f) for i, pkg in enumerate(manifest["installed_packages"]): pkg_dict = pkg["package_dict"] print(i, pkg_dict["directory"], pkg_dict["name"], pkg_dict["source"]) @TEST-END-FILE package-manager-3.0.1/testing/tests/bundle-missing-dependency0000644000175000017500000000177014565163601022436 0ustar rharha# @TEST-DOC: Create a bundle by hand with a package depending on "zkg-test-plugin" which does not exist, observe a warning during unbundling. # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # @TEST-REQUIRES: tar --version # # @TEST-EXEC: bash %INPUT # # @TEST-EXEC: zkg unbundle bundle.tar >> output # @TEST-EXEC: zkg list >> output # @TEST-EXEC: btest-diff output # @TEST-EXEC: TEST_DIFF_CANONIFIER='sed -r "s/[0-9]{2}/XX/g" | $SCRIPTS/diff-remove-abspath btest-diff' btest-diff .stderr # @TEST-EXEC: btest-diff package.log # export LOG=$(pwd)/package.log ( cd packages/foo cat >>zkg.meta <> $LOG build_command = echo "Building foo" >> $LOG depends = zkg-test-plugin >=6.0.0 EOF git commit -am 'foo: depends on zkg-test-plugin' git checkout -b origin/main ) # Create a bundle mkdir the-bundle cp -R packages/foo ./the-bundle echo -e '[bundle]\n/one/alice/foo = main' >> the-bundle/manifest.txt tar -cf bundle.tar -C ./the-bundle . package-manager-3.0.1/testing/tests/upgrade0000644000175000017500000000216614565163601017031 0ustar rharha# @TEST-EXEC: bash setup_foo # @TEST-EXEC: zkg install foo bar # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg upgrade # @TEST-EXEC: btest-diff scripts/packages/foo/__load__.zeek # @TEST-EXEC: btest-diff scripts/packages/bar/__load__.zeek @TEST-START-FILE setup_foo cd packages/foo echo 'print "foo 1.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 @TEST-END-FILE cd packages/foo echo 'print "foo 1.0.3";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.3 -m 1.0.3 echo 'print "foo 1.0.4";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.4 -m 1.0.4 echo 'print "foo master";' > __load__.zeek git commit -am 'new stuff' cd ../bar echo 'print "bar+ loaded";' > __load__.zeek git commit -am 'new stuff' echo 'print "bar 1.0.0 loaded";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar++ loaded";' > __load__.zeek git commit -am 'new stuff' package-manager-3.0.1/testing/tests/package-submodule0000644000175000017500000000127614565163601020773 0ustar rharha# @TEST-EXEC: bash setup_foo # @TEST-EXEC: zkg install foo # @TEST-EXEC: cp scripts/packages/foo/__load__.zeek foo.install # @TEST-EXEC: btest-diff foo.install # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg refresh # @TEST-EXEC: zkg upgrade # @TEST-EXEC: cp scripts/packages/foo/__load__.zeek foo.upgrade # @TEST-EXEC: btest-diff foo.upgrade @TEST-START-FILE setup_foo cd packages/foo echo 'script_dir = scripts' >> zkg.meta git submodule add $(pwd)/../bar scripts git commit -am 'add submodule' @TEST-END-FILE cd packages/bar echo 'event zeek_init() { print "bar upgrade is loaded"; }' > __load__.zeek git commit -am 'new stuff' cd ../foo/scripts git pull cd .. git commit -am 'update submodule' package-manager-3.0.1/testing/tests/bundle0000644000175000017500000000277514565163601016661 0ustar rharha# @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo baz # @TEST-EXEC: zkg install bar --version 1.0.0 # @TEST-EXEC: zkg bundle test.bundle # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg unbundle test.bundle # @TEST-EXEC: zkg list installed > snapshot.out # @TEST-EXEC: btest-diff snapshot.out # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg bundle test.bundle --manifest foo bar baz # @TEST-EXEC: zkg unbundle test.bundle # @TEST-EXEC: zkg list installed > args.out # @TEST-EXEC: btest-diff args.out # @TEST-EXEC: zkg purge # @TEST-EXEC: zkg bundle test.bundle --manifest manifest.txt # @TEST-EXEC: zkg unbundle test.bundle # @TEST-EXEC: zkg list installed > manifest.out # @TEST-EXEC: btest-diff manifest.out default_branch_name=$( cd packages/baz && git rev-parse --abbrev-ref HEAD ) echo "[bundle]" > manifest.txt echo "foo = 1.0.0" >> manifest.txt echo "bar = 1.0.1" >> manifest.txt echo "baz = ${default_branch_name}" >> manifest.txt cd packages/foo echo 'print "foo 1.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "foo 1.0.1";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "foo 1.0.2";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 cd ../bar echo 'print "bar 1.0.0";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.0 -m 1.0.0 echo 'print "bar 1.0.1";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.1 -m 1.0.1 echo 'print "bar 1.0.2";' > __load__.zeek git commit -am 'new stuff' git tag -a 1.0.2 -m 1.0.2 package-manager-3.0.1/testing/tests/aliases-conflict0000644000175000017500000000212214565163601020612 0ustar rharha# @TEST-DOC: Ensure alias conflicts are reported and cause package installation failures. # # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo >>out 2>&1 # @TEST-EXEC-FAIL: zkg install bar >>out 2>&1 # @TEST-EXEC-FAIL: zkg install baz >>out 2>&1 # @TEST-EXEC-FAIL: zkg install corge >>out 2>&1 # @TEST-EXEC: cp scripts/packages/packages.zeek packages.zeek.1 # # Remove foo, then install baz and corge # Now, remove foo, install baz and corge and ensure foo cannot be installed thereafter. # @TEST-EXEC: zkg remove foo >>out 2>&1 # @TEST-EXEC: zkg install baz >>out 2>&1 # @TEST-EXEC: zkg install corge >>out 2>&1 # @TEST-EXEC-FAIL: zkg install foo >>out 2>&1 # @TEST-EXEC: cp scripts/packages/packages.zeek packages.zeek.2 # @TEST-EXEC: btest-diff out # @TEST-EXEC: btest-diff packages.zeek.1 # @TEST-EXEC: btest-diff packages.zeek.2 (cd packages/foo && echo 'aliases = bar' >> zkg.meta && git commit -n -am 'new stuff') (cd packages/baz && echo 'aliases = foo' >> zkg.meta && git commit -n -am 'new stuff') (cd packages/corge && echo 'aliases = bar' >> zkg.meta && git commit -n -am 'new stuff') package-manager-3.0.1/testing/tests/aliases-bad0000644000175000017500000000065414565163601017547 0ustar rharha # @TEST-EXEC: bash %INPUT # @TEST-EXEC-FAIL: zkg install foo # @TEST-EXEC: TEST_DIFF_CANONIFIER='grep ^error' btest-diff .stderr cd packages/foo echo 'aliases = ../../../../../../../../bad' >> zkg.meta git commit -am 'new stuff' # @TEST-START-NEXT cd packages/foo echo 'aliases = .hidden' >> zkg.meta git commit -am 'new stuff' # @TEST-START-NEXT cd packages/foo echo 'aliases = /aaa' >> zkg.meta git commit -am 'new stuff' package-manager-3.0.1/testing/tests/install-extra0000644000175000017500000000057414565163601020172 0ustar rharha# Install from an extra source # @TEST-EXEC: zkg --extra-source two=`pwd`/sources/two search quux > search.out # @TEST-EXEC: grep -q -m 1 two/eve/quux search.out # scripts/zkg doesn't add --force for us, do it by hand # @TEST-EXEC: zkg --extra-source two=`pwd`/sources/two install quux --force # @TEST-EXEC: zkg list > list.out # @TEST-EXEC: grep -q -m 1 'two/eve/quux' list.out package-manager-3.0.1/testing/tests/search-extra0000644000175000017500000000177114565163601017771 0ustar rharha# First search for corge in source one default branch (i.e. 'master' or 'main') # # @TEST-EXEC: zkg search corge > search.out # @TEST-EXEC: grep -q -m 1 'one/bob/corge' search.out # Now overwrite source one with the branch # @TEST-EXEC: zkg --extra-source one=`pwd`/sources/one@drop-corge search corge > search.out # @TEST-EXEC: grep -m 1 'no matches' search.out # Make sure grault is still there # @TEST-EXEC: zkg --extra-source one=`pwd`/sources/one@drop-corge search grault > search.out # @TEST-EXEC: grep -m 1 'one/bob/grault' search.out # Search for quux without source two # @TEST-EXEC: zkg search quux > search.out # @TEST-EXEC: grep -m 1 'no matches' search.out # # Now in source two with explicit branch (i.e. using the default of 'master' or 'main') # @TEST-EXEC: ( cd $(pwd)/sources/two && git rev-parse --abbrev-ref HEAD ) > default_branch_name # @TEST-EXEC: zkg --extra-source two=`pwd`/sources/two@$(cat default_branch_name) search quux > search.out # @TEST-EXEC: grep -m 1 'two/eve/quux' search.out package-manager-3.0.1/testing/tests/search-config0000644000175000017500000000103214565163601020101 0ustar rharha# Tests that using a branch works in the config. # # @TEST-EXEC: bash %INPUT echo "\ [sources] one = $(pwd)/sources/one@drop-corge two = $(pwd)/sources/two [paths] state_dir = $(pwd)/state script_dir = $(pwd)/scripts plugin_dir = $(pwd)/plugins " >> my.config # No more corge to be found # @TEST-EXEC: zkg --config my.config search corge > search.out # @TEST-EXEC: grep -m 1 'no matches' search.out # quux is in source two # @TEST-EXEC: zkg --config my.config search quux > search.out # @TEST-EXEC: grep -m 1 'two/eve/quux' search.out package-manager-3.0.1/testing/tests/builtin-spicy-bundle0000644000175000017500000000167214565163601021445 0ustar rharha# @TEST-DOC: Bundle foo which depends spicy-plugin (built-in) and check the created manifest.txt # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # @TEST-REQUIRES: tar --version # # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo > output # @TEST-EXEC: zkg bundle bundle.tar --manifest foo >> output # @TEST-EXEC: zkg unbundle bundle.tar >> output # @TEST-EXEC: mkdir bundle && tar -C bundle -xf bundle.tar # @TEST-EXEC: TEST_DIFF_CANONIFIER='sed -r "s/(.*)=[0-9]+\.[0-9]+\.[0-9]+/\1=X.X.X/g" | $SCRIPTS/diff-remove-abspath btest-diff' btest-diff bundle/manifest.txt # @TEST-EXEC: btest-diff output # @TEST-EXEC: btest-diff .stderr # @TEST-EXEC: btest-diff package.log # export LOG=$(pwd)/package.log ( cd packages/foo cat >>zkg.meta <> $LOG build_command = echo "Building foo" >> $LOG depends = spicy-plugin >=6.0.0 EOF git commit -am 'foo: depends on spicy-plugin' ) package-manager-3.0.1/testing/tests/metadata-aliases0000644000175000017500000000045314565163601020576 0ustar rharha # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo # @TEST-EXEC: btest-diff scripts/foo/__load__.zeek # @TEST-EXEC: btest-diff scripts/foo2/__load__.zeek # @TEST-EXEC: btest-diff scripts/foo3/__load__.zeek cd packages/foo echo 'aliases = foo foo2 foo3' >> zkg.meta git commit -am 'new stuff' package-manager-3.0.1/testing/tests/builtin-spicy0000644000175000017500000000105114565163601020165 0ustar rharha# @TEST-DOC: Ensure a package that requires spicy-plugin installs when spicy-plugin is built-in # # @TEST-REQUIRES: zeek -e 'exit(Version::at_least("6.0.0") ? 0 : 1)' # # @TEST-EXEC: bash %INPUT # @TEST-EXEC: zkg install foo > output # @TEST-EXEC: btest-diff output # @TEST-EXEC: btest-diff package.log # export LOG=$(pwd)/package.log ( cd packages/foo cat >>zkg.meta <> $LOG build_command = echo "Building foo" >> $LOG depends = spicy-plugin * EOF git commit -am 'foo: depends on spicy-plugin' ) package-manager-3.0.1/testing/tests/template-info0000644000175000017500000000062714565163601020146 0ustar rharha# This test verifies template info retrieval via "zkg template info". # @TEST-EXEC: zkg template info --json --jsonpretty 4 $TEMPLATES/foo >output.json # @TEST-EXEC: btest-diff output.json # @TEST-EXEC: zkg template info $TEMPLATES/foo >output.plain # @TEST-EXEC: btest-diff output.plain # @TEST-EXEC: zkg template info --json --jsonpretty 4 templates/foo >output.git # @TEST-EXEC: btest-diff output.git package-manager-3.0.1/zkg.config0000644000175000017500000000756214565163601014627 0ustar rharha# This is an example config file for zkg to explain what # settings are possible as well as their default values. # The order of precedence for how zkg finds/reads config files: # # (1) zkg --configfile=/path/to/custom/config # (2) the ZKG_CONFIG_FILE environment variable # (3) a config file located at $HOME/.zkg/config # (4) if none of the above exist, then zkg uses builtin/default # values for all settings shown below [sources] # The default package source repository from which zkg fetches # packages. The default source may be removed, changed, or # additional sources may be added as long as they use a unique key # and a value that is a valid git URL. The git URL may also use a # suffix like "@branch-name" where "branch-name" is the name of a real # branch to checkout (as opposed to the default branch, which is typically # "main" or "master"). You can override the package source zkg puts # in new config files (e.g. "zkg autoconfig") by setting the # ZKG_DEFAULT_SOURCE environment variable. zeek = https://github.com/zeek/packages [paths] # Directory where source repositories are cloned, packages are # installed, and other package manager state information is # maintained. If left blank or with --user this defaults to # $HOME/.zkg. In Zeek-bundled installations, it defaults to # /var/lib/zkg/. state_dir = # The directory where package scripts are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "script_dir" option of each package's zkg.meta # (or legacy bro-pkg.meta) file there. # If left blank or with --user this defaults to /script_dir. # In Zeek-bundled installations, it defaults to # /share/zeek/site. # If you decide to change this location after having already # installed packages, zkg will automatically relocate them # the next time you run any zkg command. script_dir = # The directory where package plugins are copied upon installation. # A subdirectory named "packages" is always created within the # specified path and the package manager will copy the directory # specified by the "plugin_dir" option of each package's zkg.meta # (or legacy bro-pkg.meta) file there. # If left blank or with --user this defaults to /plugin_dir. # In Zeek-bundled installations, it defaults to # /lib/zeek/plugins. # If you decide to change this location after having already # installed packages, zkg will automatically relocate them # the next time you run any zkg command. plugin_dir = # The directory where executables from packages are linked into upon # installation. If left blank or with --user this defaults to /bin. # In Zeek-bundled installations, it defaults to /bin. # If you decide to change this location after having already # installed packages, zkg will automatically relocate them # the next time you run any zkg command. bin_dir = # The directory containing Zeek distribution source code. This is only # needed when installing packages that contain Zeek plugins that are # not pre-built. This value is generally not needed by most users other # than plugin developers anymore. zeek_dist = [templates] # The URL of the package template repository that the "zkg create" command # will instantiate by default. default = https://github.com/zeek/package-template [user_vars] # For any key in this section that is matched for value interpolation # in a package's zkg.meta (or legacy bro-pkg.meta) file, the corresponding # value is substituted during execution of the package's `build_command`. # This section is typically automatically populated with the # the answers supplied during package installation prompts # and, as a convenience feature, used to recall the last-used settings # during subsequent operations (e.g. upgrades) on the same package. package-manager-3.0.1/.github/0000755000175000017500000000000014565163601014173 5ustar rharhapackage-manager-3.0.1/.github/workflows/0000755000175000017500000000000014565163601016230 5ustar rharhapackage-manager-3.0.1/.github/workflows/test.yml0000644000175000017500000000440214565163601017732 0ustar rharhaname: Test and upload Python package on: pull_request: push: branches: [master] tags: - 'v*' - '!v*-dev' env: ZEEKROOT: /usr/local/zeek jobs: test: strategy: fail-fast: false matrix: # Test Zeek LTS, latest, and nightly, and # don't fail for the nightly one. include: - repo: zeek version: latest continue_on_error: false - repo: zeek version: lts continue_on_error: false - repo: zeek-dev version: latest continue_on_error: true runs-on: ubuntu-latest continue-on-error: ${{ matrix.continue_on_error }} container: image: zeek/${{ matrix.repo }}:${{ matrix.version }} steps: - name: Install build environment run: | apt-get update apt-get install -y --no-install-recommends cmake g++ libssl-dev libpcap-dev make - name: Remove zkg installation # Rule out confusion between test environment and pre-existing zkg: run: | rm $ZEEKROOT/bin/zkg $ZEEKROOT/share/man/man1/zkg.1 rm -r $ZEEKROOT/etc/zkg rm -r $ZEEKROOT/lib/zeek/python/zeekpkg rm -r $ZEEKROOT/var/lib/zkg - uses: actions/checkout@v4 - name: Run unit tests run: btest -j -A -d -c testing/btest.cfg - uses: actions/upload-artifact@v4 if: failure() with: name: btest-${{ matrix.repo }}-${{ matrix.version }} path: testing/.tmp/ upload: runs-on: ubuntu-latest needs: [test] if: github.repository == 'zeek/package-manager' && github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') steps: - uses: actions/checkout@v4 - name: Check release version # This fails e.g. if VERSION contains a dev commits suffix, # since we don't want to push these to PyPI. Accepts two- # and three-component version numbers (e.g. 1.0 and 1.0.1). run: | grep -E -x '[0-9]+\.[0-9]+(\.[0-9]+)?' VERSION - name: Build wheel run: | make dist - name: Upload to PyPI uses: pypa/gh-action-pypi-publish@release/v1 with: user: __token__ password: ${{ secrets.PYPI_API_TOKEN }} package-manager-3.0.1/.github/workflows/pre-commit.yml0000644000175000017500000000042314565163601021026 0ustar rharhaname: pre-commit on: pull_request: push: branches: [master] jobs: pre-commit: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v4 with: python-version: '3.7' - uses: pre-commit/action@v3.0.0 package-manager-3.0.1/VERSION0000644000175000017500000000000614565163601013677 0ustar rharha3.0.1 package-manager-3.0.1/Makefile0000644000175000017500000000141414565163601014273 0ustar rharhaVERSION=`cat VERSION` .PHONY: all all: .PHONY: doc doc: man html .PHONY: man man: (cd doc && make man && mkdir -p man && cp _build/man/zkg.1 man) .PHONY: html html: (cd doc && make html) .PHONY: livehtml livehtml: (cd doc && make livehtml) .PHONY: test test: @( cd testing && make ) .PHONY: dist dist: python3 setup.py bdist_wheel .PHONY: upload upload: twine-check dist twine upload -u zeek dist/zkg-$(VERSION)-py2.py3-none-any.whl .PHONY: twine-check twine-check: @type twine > /dev/null 2>&1 || \ { \ echo "Uploading to PyPi requires 'twine' and it's not found in PATH."; \ echo "Install it and/or make sure it is in PATH."; \ echo "E.g. you could use the following command to install it:"; \ echo "\tpip3 install twine"; \ echo ; \ exit 1; \ } package-manager-3.0.1/zeekpkg/0000755000175000017500000000000014565163601014273 5ustar rharhapackage-manager-3.0.1/zeekpkg/uservar.py0000644000175000017500000001257114565163601016342 0ustar rharha""" A module for zkg's notion of "user variables": named values required by packages that the user can provide in a variety of ways, including responses to zkg's input prompting. """ import os import re import readline def slugify(string): """Returns file-system-safe, lower-case version of the input string. Any character sequence outside of ``[a-zA-Z0-9_]+`` gets replaced by a single underscore. If the variable has no value or the value is an empty string, returns the given default. """ return re.sub(r"[^\w]+", "_", string, flags=re.ASCII).lower() def _rlinput(prompt, prefill=""): """Variation of input() that supports pre-filling a value.""" readline.set_startup_hook(lambda: readline.insert_text(prefill)) try: return input(prompt) finally: readline.set_startup_hook() class UserVar: """A class representing a single user variable. User variables have a name and an optional description. They resolve to a value using a cascade of mechanisms, including command-line arguments (via --user-var), environment variables, cached previous values, and user input. They may come with a default value. """ def __init__(self, name, val=None, default=None, desc=None): self._name = name self._desc = desc or "" self._val = val self._default = default if default is not None else val def name(self): return self._name def desc(self): return self._desc def set(self, val): self._val = val def val(self, fallback=None): return self._val if self._val is not None else fallback def default(self): return self._default def resolve(self, name, config, user_var_args=None, force=False): """Populates user variables with updated values and returns them. This function resolves the variable in the following order: (1) Use any value provided on the command line via --user-var (2) If force is not used, prompt the user for input (3) use an environment variables of the same name, (4) retrieve from the provided config parser's "user_vars" section, (5) use the default value of the user variable. The resolved value is stored with the instance (to be retrieved via .val() in the future) and also returned. Args: name (str): the requesting entity, e.g. a package name config (configparser.ConfigParser): the zkg configuration user_var_args (list of UserVar): user-var instances provided via command line force (bool): whether to skip prompting for input Returns: str: the resulting variable value Raises: ValueError: when we couldn't produce a value for the variable """ val = None source = None if user_var_args: for uvar in user_var_args: if uvar.name() == self._name: val = uvar.val() source = "command line" break if val is None: val = os.environ.get(self._name) if val: source = "environment" if source: print( '"{}" will use value of "{}" ({}) from {}: {}'.format( name, self._name, self._desc, source, val ) ) self._val = val return val if val is None: # Try to re-use a cached value in the subsequent prompt val = config.get("user_vars", self._name, fallback=self._default) if force: if val is None: raise ValueError(self._name) self._val = val return val desc = " (" + self._desc + ")" if self._desc else "" print(f'"{name}" requires a "{self._name}" value{desc}: ') self._val = _rlinput(self._name + ": ", val) return self._val @staticmethod def parse_arg(arg): """Parser for NAME=VAL format string used in command-line args.""" try: name, val = arg.split("=", 1) return UserVar(name, val=val) except ValueError as error: raise ValueError( f'invalid user var argument "{arg}", must be NAME=VAR' ) from error @staticmethod def parse_dict(metadata_dict): """Returns list of UserVars from the metadata's 'user_vars' field. Args: metadata_dict (dict of str->str): the input metadata, e.g. from a configparser entry value. Returns: list of UserVar. If the 'user_vars' field is not present, an empty list is returned. If malformed, returns None. """ text = metadata_dict.get("user_vars") if not text: return [] rval = [] text = text.strip() # Example: LIBRDKAFKA_ROOT [/usr] "Path to librdkafka installation" entries = re.split(r'(\w+\s+\[.*\]\s+".*")\s+', text) entries = list(filter(None, entries)) for entry in entries: mob = re.match(r'(\w+)\s+\[(.*)\]\s+"(.*)"', entry) if not mob: return None groups = mob.groups() if len(groups) != 3: return None rval.append(UserVar(groups[0], val=groups[1], desc=groups[2])) return rval package-manager-3.0.1/zeekpkg/manager.py0000644000175000017500000037427214565163601016276 0ustar rharha""" A module defining the main Zeek Package Manager interface which supplies methods to interact with and operate on Zeek packages. """ import configparser import copy import filecmp import json import os import pathlib import re import shutil import subprocess import sys import tarfile try: from urllib.parse import urlparse except ImportError: from urlparse import urlparse import git import semantic_version as semver from collections import deque from ._util import ( make_dir, delete_path, make_symlink, copy_over_path, get_zeek_info, git_default_branch, git_checkout, git_clone, git_pull, git_version_tags, is_sha1, get_zeek_version, std_encoding, find_program, read_zeek_config_line, normalize_version_tag, configparser_section_dict, safe_tarfile_extractall, ) from .source import AGGREGATE_DATA_FILE, Source from .package import ( BUILTIN_SOURCE, BUILTIN_SCHEME, METADATA_FILENAME, LEGACY_METADATA_FILENAME, TRACKING_METHOD_VERSION, TRACKING_METHOD_BRANCH, TRACKING_METHOD_COMMIT, PLUGIN_MAGIC_FILE, PLUGIN_MAGIC_FILE_DISABLED, LEGACY_PLUGIN_MAGIC_FILE, LEGACY_PLUGIN_MAGIC_FILE_DISABLED, name_from_path, aliases, canonical_url, is_valid_name as is_valid_package_name, make_builtin_package, Package, PackageInfo, PackageStatus, InstalledPackage, PackageVersion, ) from .uservar import ( UserVar, ) from . import ( __version__, LOG, ) class Stage: def __init__(self, manager, state_dir=None): self.manager = manager if state_dir: self.state_dir = state_dir self.clone_dir = os.path.join(self.state_dir, "clones") self.script_dir = os.path.join(self.state_dir, "scripts", "packages") self.plugin_dir = os.path.join(self.state_dir, "plugins", "packages") self.bin_dir = os.path.join(self.state_dir, "bin") else: # Stages not given a test directory are essentially a shortcut to # standard functionality; this doesn't require all directories: self.state_dir = None self.clone_dir = manager.package_clonedir self.script_dir = manager.script_dir self.plugin_dir = manager.plugin_dir self.bin_dir = manager.bin_dir def populate(self): # If we're staging to a temporary location, blow anything existing there # away first. if self.state_dir: delete_path(self.state_dir) make_dir(self.clone_dir) make_dir(self.script_dir) make_dir(self.plugin_dir) make_dir(self.bin_dir) # To preserve %(package_base)s functionality in build/test commands # during staging in testing folders, we need to provide one location # that combines the existing installed packages, plus any under test, # with the latter overriding any already installed ones. We symlink the # real install folders into the staging one. The subsequent cloning of # the packages under test will remove those links as needed. if self.state_dir: with os.scandir(self.manager.package_clonedir) as it: for entry in it: if not entry.is_dir(): continue make_symlink(entry.path, os.path.join(self.clone_dir, entry.name)) def get_subprocess_env(self): zeekpath = os.environ.get("ZEEKPATH") pluginpath = os.environ.get("ZEEK_PLUGIN_PATH") if not (zeekpath and pluginpath): zeek_config = find_program("zeek-config") if zeek_config: cmd = subprocess.Popen( [zeek_config, "--zeekpath", "--plugin_dir"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True, ) line1 = read_zeek_config_line(cmd.stdout) line2 = read_zeek_config_line(cmd.stdout) if not zeekpath: zeekpath = line1 if not pluginpath: pluginpath = line2 else: return None, 'no "zeek-config" found in PATH' zeekpath = os.path.dirname(self.script_dir) + os.pathsep + zeekpath pluginpath = os.path.dirname(self.plugin_dir) + os.pathsep + pluginpath env = os.environ.copy() env["PATH"] = self.bin_dir + os.pathsep + os.environ.get("PATH", "") env["ZEEKPATH"] = zeekpath env["ZEEK_PLUGIN_PATH"] = pluginpath return env, "" class Manager: """A package manager object performs various operations on packages. It uses a state directory and a manifest file within it to keep track of package sources, installed packages and their statuses. Attributes: sources (dict of str -> :class:`.source.Source`): dictionary package sources keyed by the name given to :meth:`add_source()` installed_pkgs (dict of str -> :class:`.package.InstalledPackage`): a dictionary of installed packaged keyed on package names (the last component of the package's git URL) zeek_dist (str): path to the Zeek source code distribution. This is needed for packages that contain Zeek plugins that need to be built from source code. state_dir (str): the directory where the package manager will a maintain manifest file, package/source git clones, and other persistent state the manager needs in order to operate user_vars (dict of str -> str): dictionary of key-value pairs where the value will be substituted into package build commands in place of the key. backup_dir (str): a directory where the package manager will store backup files (e.g. locally modified package config files) log_dir (str): a directory where the package manager will store misc. logs files (e.g. package build logs) scratch_dir (str): a directory where the package manager performs miscellaneous/temporary file operations script_dir (str): the directory where the package manager will copy each installed package's `script_dir` (as given by its :file:`zkg.meta` or :file:`bro-pkg.meta`). Each package gets a subdirectory within `script_dir` associated with its name. plugin_dir (str): the directory where the package manager will copy each installed package's `plugin_dir` (as given by its :file:`zkg.meta` or :file:`bro-pkg.meta`). Each package gets a subdirectory within `plugin_dir` associated with its name. bin_dir (str): the directory where the package manager will link executables into that are provided by an installed package through `executables` (as given by its :file:`zkg.meta` or :file:`bro-pkg.meta`) source_clonedir (str): the directory where the package manager will clone package sources. Each source gets a subdirectory associated with its name. package_clonedir (str): the directory where the package manager will clone installed packages. Each package gets a subdirectory associated with its name. package_testdir (str): the directory where the package manager will run tests. Each package gets a subdirectory associated with its name. manifest (str): the path to the package manager's manifest file. This file maintains a list of installed packages and their status. autoload_script (str): path to a Zeek script named :file:`packages.zeek` that the package manager maintains. It is a list of ``@load`` for each installed package that is marked as loaded (see :meth:`load()`). autoload_package (str): path to a Zeek :file:`__load__.zeek` script which is just a symlink to `autoload_script`. It's always located in a directory named :file:`packages`, so as long as :envvar:`ZEEKPATH` is configured correctly, ``@load packages`` will load all installed packages that have been marked as loaded. """ def __init__( self, state_dir, script_dir, plugin_dir, zeek_dist="", user_vars=None, bin_dir="", ): """Creates a package manager instance. Args: state_dir (str): value to set the `state_dir` attribute to script_dir (str): value to set the `script_dir` attribute to plugin_dir (str): value to set the `plugin_dir` attribute to zeek_dist (str): value to set the `zeek_dist` attribute to user_vars (dict of str -> str): key-value pair substitutions for use in package build commands. bin_dir (str): value to set the `bin_dir` attribute to. If empty/nil value, defaults to setting `bin_dir` attribute to `/bin`. Raises: OSError: when a package manager state directory can't be created IOError: when a package manager state file can't be created """ LOG.debug("init Manager version %s", __version__) self.sources = {} self.installed_pkgs = {} self._builtin_packages = None # Cached Zeek built-in packages. self._builtin_packages_discovered = False # Flag if discovery even worked. self.zeek_dist = zeek_dist self.state_dir = state_dir self.user_vars = {} if user_vars is None else user_vars self.backup_dir = os.path.join(self.state_dir, "backups") self.log_dir = os.path.join(self.state_dir, "logs") self.scratch_dir = os.path.join(self.state_dir, "scratch") self._script_dir = script_dir self.script_dir = os.path.join(script_dir, "packages") self._plugin_dir = plugin_dir self.plugin_dir = os.path.join(plugin_dir, "packages") self.bin_dir = bin_dir or os.path.join(self.state_dir, "bin") self.source_clonedir = os.path.join(self.state_dir, "clones", "source") self.package_clonedir = os.path.join(self.state_dir, "clones", "package") self.package_testdir = os.path.join(self.state_dir, "testing") self.manifest = os.path.join(self.state_dir, "manifest.json") self.autoload_script = os.path.join(self.script_dir, "packages.zeek") self.autoload_package = os.path.join(self.script_dir, "__load__.zeek") make_dir(self.state_dir) make_dir(self.log_dir) make_dir(self.scratch_dir) make_dir(self.source_clonedir) make_dir(self.package_clonedir) make_dir(self.script_dir) make_dir(self.plugin_dir) make_dir(self.bin_dir) _create_readme(os.path.join(self.script_dir, "README")) _create_readme(os.path.join(self.plugin_dir, "README")) if not os.path.exists(self.manifest): self._write_manifest() prev_script_dir, prev_plugin_dir, prev_bin_dir = self._read_manifest() # Place all Zeek built-in packages into installed packages. for info in self.discover_builtin_packages(): self.installed_pkgs[info.package.name] = InstalledPackage( package=info.package, status=info.status ) refresh_bin_dir = False # whether we need to updates link in bin_dir relocating_bin_dir = False # whether bin_dir has relocated need_manifest_update = False if os.path.realpath(prev_script_dir) != os.path.realpath(self.script_dir): LOG.info("relocating script_dir %s -> %s", prev_script_dir, self.script_dir) if os.path.exists(prev_script_dir): delete_path(self.script_dir) shutil.move(prev_script_dir, self.script_dir) prev_zeekpath = os.path.dirname(prev_script_dir) for pkg_name in self.installed_pkgs: old_link = os.path.join(prev_zeekpath, pkg_name) new_link = os.path.join(self.zeekpath(), pkg_name) if os.path.lexists(old_link): LOG.info("moving package link %s -> %s", old_link, new_link) shutil.move(old_link, new_link) else: LOG.info("skip moving package link %s -> %s", old_link, new_link) need_manifest_update = True refresh_bin_dir = True if os.path.realpath(prev_plugin_dir) != os.path.realpath(self.plugin_dir): LOG.info("relocating plugin_dir %s -> %s", prev_plugin_dir, self.plugin_dir) if os.path.exists(prev_plugin_dir): delete_path(self.plugin_dir) shutil.move(prev_plugin_dir, self.plugin_dir) need_manifest_update = True refresh_bin_dir = True if prev_bin_dir and os.path.realpath(prev_bin_dir) != os.path.realpath( self.bin_dir ): LOG.info("relocating bin_dir %s -> %s", prev_bin_dir, self.bin_dir) need_manifest_update = True refresh_bin_dir = True relocating_bin_dir = True if refresh_bin_dir: self._refresh_bin_dir(self.bin_dir) if relocating_bin_dir: self._clear_bin_dir(prev_bin_dir) try: # We try to remove the old bin_dir. That may not succeed in case # it wasn't actually managed by us, but that's ok. os.rmdir(prev_bin_dir) except os.error: pass if need_manifest_update: self._write_manifest() self._write_autoloader() make_symlink("packages.zeek", self.autoload_package) def _write_autoloader(self): """Write the :file:`packages.zeek` loader script. Raises: IOError: if :file:`packages.zeek` loader script cannot be written """ with open(self.autoload_script, "w") as f: content = ( "# WARNING: This file is managed by zkg.\n" "# Do not make direct modifications here.\n" ) for ipkg in self.loaded_packages(): if self.has_scripts(ipkg): content += f"@load ./{ipkg.package.name}\n" f.write(content) def _write_plugin_magic(self, ipkg): """Enables/disables any Zeek plugin included with a package. Zeek's plugin code scans its plugin directories for __zeek_plugin__ magic files, which indicate presence of a plugin directory. When this file does not exist, Zeek does not recognize a plugin. When we're loading a package, this function renames an existing __zeek_plugin__.disabled file to __zeek_plugin__, and vice versa when we're unloading a package. When the package doesn't include a plugin, or when the plugin directory already contains a correctly named magic file, this function does nothing. Until Zeek 6.1, the magic file was named __bro_plugin__. zkg implements a fallback for recognizing the older name so that newer zkg versions continue to work with older Zeek versions for some time longer. """ package_dir = pathlib.Path(self.plugin_dir) / ipkg.package.name magic_paths_enabled = [ package_dir / PLUGIN_MAGIC_FILE, package_dir / LEGACY_PLUGIN_MAGIC_FILE, ] magic_paths_disabled = [ package_dir / PLUGIN_MAGIC_FILE_DISABLED, package_dir / LEGACY_PLUGIN_MAGIC_FILE_DISABLED, ] for path_enabled, path_disabled in zip( magic_paths_enabled, magic_paths_disabled ): if ipkg.status.is_loaded: if path_disabled.exists(): try: path_disabled.rename(path_enabled) except OSError as exception: LOG.error( "could not enable plugin: %s %s", type(exception).__name__, exception, ) else: if path_enabled.exists(): try: path_enabled.rename(path_disabled) except OSError as exception: LOG.error( "could not disable plugin: %s %s", type(exception).__name__, exception, ) def _read_manifest(self): """Read the manifest file containing the list of installed packages. Returns: tuple: (previous script_dir, previous plugin_dir) Raises: IOError: when the manifest file can't be read """ with open(self.manifest) as f: data = json.load(f) version = data["manifest_version"] pkg_list = data["installed_packages"] self.installed_pkgs = {} for dicts in pkg_list: pkg_dict = dicts["package_dict"] status_dict = dicts["status_dict"] pkg_name = pkg_dict["name"] if version == 0 and "index_data" in pkg_dict: del pkg_dict["index_data"] pkg_dict["canonical"] = True pkg = Package(**pkg_dict) status = PackageStatus(**status_dict) self.installed_pkgs[pkg_name] = InstalledPackage(pkg, status) return data["script_dir"], data["plugin_dir"], data.get("bin_dir", None) def _write_manifest(self): """Writes the manifest file containing the list of installed packages. Raises: IOError: when the manifest file can't be written """ pkg_list = [] for _, installed_pkg in self.installed_pkgs.items(): if installed_pkg.is_builtin(): continue pkg_list.append( { "package_dict": installed_pkg.package.__dict__, "status_dict": installed_pkg.status.__dict__, } ) data = { "manifest_version": 1, "script_dir": self.script_dir, "plugin_dir": self.plugin_dir, "bin_dir": self.bin_dir, "installed_packages": pkg_list, } with open(self.manifest, "w") as f: json.dump(data, f, indent=2, sort_keys=True) def zeekpath(self): """Return the path where installed package scripts are located. This path can be added to :envvar:`ZEEKPATH` for interoperability with Zeek. """ return os.path.dirname(self.script_dir) def zeek_plugin_path(self): """Return the path where installed package plugins are located. This path can be added to :envvar:`ZEEK_PLUGIN_PATH` for interoperability with Zeek. """ return os.path.dirname(self.plugin_dir) def add_source(self, name, git_url): """Add a git repository that acts as a source of packages. Args: name (str): a short name that will be used to reference the package source. git_url (str): the git URL of the package source Returns: str: empty string if the source is successfully added, else the reason why it failed. """ if name == BUILTIN_SOURCE: return f"{name} is a reserved source name" if name in self.sources: existing_source = self.sources[name] if existing_source.git_url == git_url: LOG.debug('duplicate source "%s"', name) return True return "source already exists with different URL: {}".format( existing_source.git_url ) clone_path = os.path.join(self.source_clonedir, name) # Support @ in the path to denote the "version" to checkout version = None # Prepend 'ssh://' and replace the first ':' with '/' if git_url # looks like a scp-like URL, e.g. git@github.com:user/repo.git. # urlparse will otherwise parse everything into path and the @ # is confusing the versioning logic. Note that per the git-clone # docs git recognizes scp-style URLs only when there are no slashes # before the first colon. colonidx, slashidx = git_url.find(":"), git_url.find("/") if ( "://" not in git_url and colonidx > 0 and (slashidx == -1 or slashidx > colonidx) ): parse_result = urlparse("ssh://" + git_url.replace(":", "/", 1)) else: parse_result = urlparse(git_url) if parse_result.path and "@" in parse_result.path: git_url, version = git_url.rsplit("@", 1) try: source = Source( name=name, clone_path=clone_path, git_url=git_url, version=version, ) except git.exc.GitCommandError as error: LOG.warning("failed to clone git repo: %s", error) return "failed to clone git repo" else: self.sources[name] = source return "" def source_packages(self): """Return a list of :class:`.package.Package` within all sources.""" rval = [] for _, source in self.sources.items(): rval += source.packages() return rval def discover_builtin_packages(self): """ Discover packages included in Zeek for dependency resolution. This is using Zeek's ``--build-info`` flag and specifically the ``zkg.provides`` entry it contains. Requires Zeek 6.0 and later. Returns: list of :class:`.package.BuiltinPackage`: List of built-in packages. """ if self._builtin_packages is not None: return self._builtin_packages self._builtin_packages = [] try: zeek_executable = get_zeek_info().zeek except LookupError as e: LOG.warning("unable to discover builtin-packages: %s", str(e)) return self._builtin_packages try: build_info_str = subprocess.check_output( [zeek_executable, "--build-info"], stderr=subprocess.DEVNULL, timeout=10 ) build_info = json.loads(build_info_str) except subprocess.CalledProcessError: # Not a warning() due to being a bit noisy. LOG.info("unable to discover built-in packages - requires Zeek 6.0") return self._builtin_packages except json.JSONDecodeError as e: LOG.error("unable to parse Zeek's build info output: %s", str(e)) return self._builtin_packages if "zkg" not in build_info or "provides" not in build_info["zkg"]: LOG.warning("missing zkg.provides entry in zeek --build-info output") return self._builtin_packages self._builtin_packages_discovered = True for p in build_info["zkg"]["provides"]: name, version = p.get("name"), p.get("version") commit = p.get("commit") if not name or not version: LOG.warning("zkg.provides entry missing name or version: %s", repr(p)) continue orig_version = version # The "version" field may not be semantic version compatible. # For example, 1.4.2-68 is parsed as prerelease 68 of 1.4.2, but # from update-changes/git describe, it's 68 commits after 1.4.2. # Deal with that by stripping -68, but leave -rc1 or -dev alone. m = re.match(r"([0-9]+\.[0-9]+\.[0-9]+)-[0-9]+", version) if m: version = m.group(1) LOG.debug( "found built-in package %s with version %s (%s)", name, version, orig_version, ) self._builtin_packages.append( make_builtin_package( name=name, current_version=version, current_hash=commit, ) ) return self._builtin_packages def find_builtin_package(self, pkg_path): """ Find a builtin plugin that matches ``pkg_path``. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. Returns: PackageInfo: PackageInfo instance representing a builtin package matching ``pkg_path``. """ pkg_name = name_from_path(pkg_path) for info in self.discover_builtin_packages(): if info.package.matches_path(pkg_name): return info return None def installed_packages(self): """Return list of :class:`.package.InstalledPackage`.""" return [ipkg for _, ipkg in sorted(self.installed_pkgs.items())] def installed_package_dependencies(self): """Return dict of 'package' -> dict of 'dependency' -> 'version'. Package-name / dependency-name / and version-requirement values are all strings. """ return { name: ipkg.package.dependencies() for name, ipkg in self.installed_pkgs.items() } def loaded_packages(self): """Return list of loaded :class:`.package.InstalledPackage`.""" rval = [] for _, ipkg in sorted(self.installed_pkgs.items()): if ipkg.status.is_loaded: rval.append(ipkg) return rval def package_build_log(self, pkg_path): """Return the path to the package manager's build log for a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". """ name = name_from_path(pkg_path) return os.path.join(self.log_dir, f"{name}-build.log") def match_source_packages(self, pkg_path): """Return a list of :class:`.package.Package` that match a given path. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". """ rval = [] canon_url = canonical_url(pkg_path) for pkg in self.source_packages(): if pkg.matches_path(canon_url): rval.append(pkg) return rval def find_installed_package(self, pkg_path): """Return an :class:`.package.InstalledPackage` if one matches the name. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". """ pkg_name = name_from_path(pkg_path) return self.installed_pkgs.get(pkg_name) def get_installed_package_dependencies(self, pkg_path): """Return a set of tuples of dependent package names and their version number if pkg_path is an installed package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". """ ipkg = self.find_installed_package(pkg_path) if ipkg: return ipkg.package.dependencies() return None def has_scripts(self, installed_pkg): """Return whether a :class:`.package.InstalledPackage` installed scripts. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to check for whether it has installed any Zeek scripts. Returns: bool: True if the package has installed Zeek scripts. """ return os.path.exists(os.path.join(self.script_dir, installed_pkg.package.name)) def has_plugin(self, installed_pkg): """Return whether a :class:`.package.InstalledPackage` installed a plugin. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to check for whether it has installed a Zeek plugin. Returns: bool: True if the package has installed a Zeek plugin. """ return os.path.exists(os.path.join(self.plugin_dir, installed_pkg.package.name)) def save_temporary_config_files(self, installed_pkg): """Return a list of temporary package config file backups. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to save temporary config file backups for. Returns: list of (str, str): tuples that describe the config files backups. The first element is the config file as specified in the package metadata (a file path relative to the package's root directory). The second element is an absolute file system path to where that config file has been copied. It should be considered temporary, so make use of it before doing any further operations on packages. """ import re metadata = installed_pkg.package.metadata config_files = re.split(r",\s*", metadata.get("config_files", "")) if not config_files: return [] pkg_name = installed_pkg.package.name clone_dir = os.path.join(self.package_clonedir, pkg_name) rval = [] for config_file in config_files: config_file_path = os.path.join(clone_dir, config_file) if not os.path.isfile(config_file_path): LOG.info( "package '%s' claims config file at '%s'," " but it does not exist", pkg_name, config_file, ) continue backup_file = os.path.join(self.scratch_dir, "tmpcfg", config_file) make_dir(os.path.dirname(backup_file)) shutil.copy2(config_file_path, backup_file) rval.append((config_file, backup_file)) return rval def modified_config_files(self, installed_pkg): """Return a list of package config files that the user has modified. Args: installed_pkg(:class:`.package.InstalledPackage`): the installed package to check for whether it has installed any Zeek scripts. Returns: list of (str, str): tuples that describe the modified config files. The first element is the config file as specified in the package metadata (a file path relative to the package's root directory). The second element is an absolute file system path to where that config file is currently installed. """ import re metadata = installed_pkg.package.metadata config_files = re.split(r",\s*", metadata.get("config_files", "")) if not config_files: return [] pkg_name = installed_pkg.package.name script_install_dir = os.path.join(self.script_dir, pkg_name) plugin_install_dir = os.path.join(self.plugin_dir, pkg_name) clone_dir = os.path.join(self.package_clonedir, pkg_name) script_dir = metadata.get("script_dir", "") plugin_dir = metadata.get("plugin_dir", "build") rval = [] for config_file in config_files: their_config_file_path = os.path.join(clone_dir, config_file) if not os.path.isfile(their_config_file_path): LOG.info( "package '%s' claims config file at '%s'," " but it does not exist", pkg_name, config_file, ) continue if config_file.startswith(plugin_dir): our_config_file_path = os.path.join( plugin_install_dir, config_file[len(plugin_dir) :] ) if not os.path.isfile(our_config_file_path): LOG.info( "package '%s' config file '%s' not found" " in plugin_dir: %s", pkg_name, config_file, our_config_file_path, ) continue elif config_file.startswith(script_dir): our_config_file_path = os.path.join( script_install_dir, config_file[len(script_dir) :] ) if not os.path.isfile(our_config_file_path): LOG.info( "package '%s' config file '%s' not found" " in script_dir: %s", pkg_name, config_file, our_config_file_path, ) continue else: # Their config file is outside script/plugin install dirs, # so no way user has it even installed, much less modified. LOG.warning( "package '%s' config file '%s' not within" " plugin_dir or script_dir", pkg_name, config_file, ) continue if not filecmp.cmp(our_config_file_path, their_config_file_path): rval.append((config_file, our_config_file_path)) return rval def backup_modified_files(self, backup_subdir, modified_files): """Creates backups of modified config files Args: modified_files(list of (str, str)): the return value of :meth:`modified_config_files()`. backup_subdir(str): the subdir of `backup_dir` in which Returns: list of str: paths indicating the backup locations. The order of the returned list corresponds directly to the order of `modified_files`. """ import time rval = [] for modified_file in modified_files: config_file = modified_file[0] config_file_dir = os.path.dirname(config_file) install_path = modified_file[1] filename = os.path.basename(install_path) backup_dir = os.path.join(self.backup_dir, backup_subdir, config_file_dir) timestamp = time.strftime(".%Y-%m-%d-%H:%M:%S") backup_path = os.path.join(backup_dir, filename + timestamp) make_dir(backup_dir) shutil.copy2(install_path, backup_path) rval.append(backup_path) return rval class SourceAggregationResults: """The return value of a call to :meth:`.Manager.aggregate_source()`. Attributes: refresh_error (str): an empty string if no overall error occurred in the "refresh" operation, else a description of what wrong package_issues (list of (str, str)): a list of reasons for failing to collect metadata per packages/repository. The first tuple element gives the repository URL in which the problem occurred and the second tuple element describes the failure. """ def __init__(self, refresh_error="", package_issues=[]): self.refresh_error = refresh_error self.package_issues = package_issues def aggregate_source(self, name, push=False): """Pull latest git info from a package source and aggregate metadata. This is like calling :meth:`refresh_source()` with the *aggregate* arguments set to True. This makes the latest pre-aggregated package metadata available or performs the aggregation locally in order to push it to the actual package source. Locally aggregated data also takes precedence over the source's pre-aggregated data, so it can be useful in the case the operator of the source does not update their pre-aggregated data at a frequent enough interval. Args: name(str): the name of the package source. E.g. the same name used as a key to :meth:`add_source()`. push (bool): whether to push local changes to the aggregated metadata to the remote package source. Returns: :class:`.Manager.SourceAggregationResults`: the results of the refresh/aggregation. """ return self._refresh_source(name, True, push) def refresh_source(self, name, aggregate=False, push=False): """Pull latest git information from a package source. This makes the latest pre-aggregated package metadata available or performs the aggregation locally in order to push it to the actual package source. Locally aggregated data also takes precedence over the source's pre-aggregated data, so it can be useful in the case the operator of the source does not update their pre-aggregated data at a frequent enough interval. Args: name(str): the name of the package source. E.g. the same name used as a key to :meth:`add_source()`. aggregate (bool): whether to perform a local metadata aggregation by crawling all packages listed in the source's index files. push (bool): whether to push local changes to the aggregated metadata to the remote package source. If the `aggregate` flag is set, the data will be pushed after the aggregation is finished. Returns: str: an empty string if no errors occurred, else a description of what went wrong. """ res = self._refresh_source(name, aggregate, push) return res.refresh_error def _refresh_source(self, name, aggregate=False, push=False): """Used by :meth:`refresh_source()` and :meth:`aggregate_source()`.""" if name not in self.sources: return self.SourceAggregationResults("source name does not exist") source = self.sources[name] LOG.debug('refresh "%s": pulling %s', name, source.git_url) aggregate_file = os.path.join(source.clone.working_dir, AGGREGATE_DATA_FILE) agg_file_ours = os.path.join(self.scratch_dir, AGGREGATE_DATA_FILE) agg_file_their_orig = os.path.join( self.scratch_dir, AGGREGATE_DATA_FILE + ".orig" ) delete_path(agg_file_ours) delete_path(agg_file_their_orig) if os.path.isfile(aggregate_file): shutil.copy2(aggregate_file, agg_file_ours) source.clone.git.reset(hard=True) source.clone.git.clean("-f", "-x", "-d") if os.path.isfile(aggregate_file): shutil.copy2(aggregate_file, agg_file_their_orig) try: source.clone.git.fetch("--recurse-submodules=yes") git_pull(source.clone) except git.exc.GitCommandError as error: LOG.error("failed to pull source %s: %s", name, error) return self.SourceAggregationResults( f"failed to pull from remote source: {error}" ) if os.path.isfile(agg_file_ours): if os.path.isfile(aggregate_file): # There's a tracked version of the file after pull. if os.path.isfile(agg_file_their_orig): # We had local modifications to the file. if filecmp.cmp(aggregate_file, agg_file_their_orig): # Their file hasn't changed, use ours. shutil.copy2(agg_file_ours, aggregate_file) LOG.debug( "aggegrate file in source unchanged, restore local one" ) else: # Their file changed, use theirs. LOG.debug("aggegrate file in source changed, discard local one") else: # File was untracked before pull and tracked after, # use their version. LOG.debug("new aggegrate file in source, discard local one") else: # They don't have the file after pulling, so restore ours. shutil.copy2(agg_file_ours, aggregate_file) LOG.debug("no aggegrate file in source, restore local one") aggregation_issues = [] if aggregate: parser = configparser.ConfigParser(interpolation=None) prev_parser = configparser.ConfigParser(interpolation=None) prev_packages = set() if os.path.isfile(aggregate_file): prev_parser.read(aggregate_file) prev_packages = set(prev_parser.sections()) agg_adds = [] agg_mods = [] agg_dels = [] for index_file in source.package_index_files(): urls = [] with open(index_file) as f: urls = [line.rstrip("\n") for line in f] for url in urls: pkg_name = name_from_path(url) clonepath = os.path.join(self.scratch_dir, pkg_name) delete_path(clonepath) try: clone = git_clone(url, clonepath, shallow=True) except git.exc.GitCommandError as error: LOG.warn( "failed to clone %s, skipping aggregation: %s", url, error ) aggregation_issues.append((url, repr(error))) continue version_tags = git_version_tags(clone) if len(version_tags): version = version_tags[-1] else: version = git_default_branch(clone) try: git_checkout(clone, version) except git.exc.GitCommandError as error: LOG.warn( 'failed to checkout branch/version "%s" of %s, ' "skipping aggregation: %s", version, url, error, ) msg = 'failed to checkout branch/version "{}": {}'.format( version, repr(error) ) aggregation_issues.append((url, msg)) continue metadata_file = _pick_metadata_file(clone.working_dir) metadata_parser = configparser.ConfigParser(interpolation=None) invalid_reason = _parse_package_metadata( metadata_parser, metadata_file ) if invalid_reason: LOG.warn( "skipping aggregation of %s: bad metadata: %s", url, invalid_reason, ) aggregation_issues.append((url, invalid_reason)) continue metadata = _get_package_metadata(metadata_parser) index_dir = os.path.dirname(index_file)[ len(self.source_clonedir) + len(name) + 2 : ] qualified_name = os.path.join(index_dir, pkg_name) parser.add_section(qualified_name) for key, value in sorted(metadata.items()): parser.set(qualified_name, key, value) parser.set(qualified_name, "url", url) parser.set(qualified_name, "version", version) if qualified_name not in prev_packages: agg_adds.append(qualified_name) else: prev_meta = configparser_section_dict( prev_parser, qualified_name ) new_meta = configparser_section_dict(parser, qualified_name) if prev_meta != new_meta: agg_mods.append(qualified_name) with open(aggregate_file, "w") as f: parser.write(f) agg_dels = list(prev_packages.difference(set(parser.sections()))) adds_str = " (" + ", ".join(sorted(agg_adds)) + ")" if agg_adds else "" mods_str = " (" + ", ".join(sorted(agg_mods)) + ")" if agg_mods else "" dels_str = " (" + ", ".join(sorted(agg_dels)) + ")" if agg_dels else "" LOG.debug( "metadata refresh: %d additions%s, %d changes%s, %d removals%s", len(agg_adds), adds_str, len(agg_mods), mods_str, len(agg_dels), dels_str, ) if push: if os.path.isfile( os.path.join(source.clone.working_dir, AGGREGATE_DATA_FILE) ): source.clone.git.add(AGGREGATE_DATA_FILE) if source.clone.is_dirty(): # There's an assumption here that the dirty state is # due to a metadata refresh. This could be incorrect # if somebody makes local modifications and then runs # the refresh without --aggregate, but it's not clear # why one would use zkg for this as opposed to git # itself. source.clone.git.commit( "--no-verify", "--message", "Update aggregated metadata." ) LOG.info('committed package source "%s" metadata update', name) source.clone.git.push("--no-verify") return self.SourceAggregationResults("", aggregation_issues) def refresh_installed_packages(self): """Fetch latest git information for installed packages. This retrieves information about outdated packages, but does not actually upgrade their installations. Raises: IOError: if the package manifest file can't be written """ for ipkg in self.installed_packages(): if ipkg.is_builtin(): LOG.debug( 'skipping refresh of built-in package "%s"', ipkg.package.name ) continue clonepath = os.path.join(self.package_clonedir, ipkg.package.name) clone = git.Repo(clonepath) LOG.debug("fetch package %s", ipkg.package.qualified_name()) try: clone.git.fetch("--recurse-submodules=yes") except git.exc.GitCommandError as error: LOG.warn( "failed to fetch package %s: %s", ipkg.package.qualified_name(), error, ) ipkg.status.is_outdated = _is_clone_outdated( clone, ipkg.status.current_version, ipkg.status.tracking_method ) self._write_manifest() def upgrade(self, pkg_path): """Upgrade a package to the latest available version. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: str: an empty string if package upgrade succeeded else an error string explaining why it failed. Raises: IOError: if the manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('upgrading "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('upgrading "%s": no matching package', pkg_path) return "no such package installed" if ipkg.status.is_pinned: LOG.info('upgrading "%s": package is pinned', pkg_path) return "package is pinned" if not ipkg.status.is_outdated: LOG.info('upgrading "%s": package not outdated', pkg_path) return "package is not outdated" clonepath = os.path.join(self.package_clonedir, ipkg.package.name) clone = git.Repo(clonepath) if ipkg.status.tracking_method == TRACKING_METHOD_VERSION: version_tags = git_version_tags(clone) return self._install(ipkg.package, version_tags[-1]) elif ipkg.status.tracking_method == TRACKING_METHOD_BRANCH: git_pull(clone) return self._install(ipkg.package, ipkg.status.current_version) elif ipkg.status.tracking_method == TRACKING_METHOD_COMMIT: # The above check for whether the installed package is outdated # also should have already caught this situation. return "package is not outdated" else: raise NotImplementedError def remove(self, pkg_path): """Remove an installed package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: bool: True if an installed package was removed, else False. Raises: IOError: if the package manifest file can't be written OSError: if the installed package's directory can't be deleted """ pkg_path = canonical_url(pkg_path) LOG.debug('removing "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('removing "%s": could not find matching package', pkg_path) return False if ipkg.is_builtin(): LOG.error('cannot remove built-in package "%s"', pkg_path) return False self.unload(pkg_path) pkg_to_remove = ipkg.package delete_path(os.path.join(self.package_clonedir, pkg_to_remove.name)) delete_path(os.path.join(self.script_dir, pkg_to_remove.name)) delete_path(os.path.join(self.plugin_dir, pkg_to_remove.name)) delete_path(os.path.join(self.zeekpath(), pkg_to_remove.name)) for alias in pkg_to_remove.aliases(): delete_path(os.path.join(self.zeekpath(), alias)) for exec in self._get_executables(pkg_to_remove.metadata): link = os.path.join(self.bin_dir, os.path.basename(exec)) if os.path.islink(link): try: LOG.debug("removing link %s", link) os.unlink(link) except os.error as err: LOG.warn("cannot remove link for %s", err) del self.installed_pkgs[pkg_to_remove.name] self._write_manifest() LOG.debug('removed "%s"', pkg_path) return True def pin(self, pkg_path): """Pin a currently installed package to the currently installed version. Pinned packages are never upgraded when calling :meth:`upgrade()`. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: :class:`.package.InstalledPackage`: None if no matching installed package could be found, else the installed package that was pinned. Raises: IOError: when the manifest file can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('pinning "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('pinning "%s": no matching package', pkg_path) return None if ipkg.status.is_pinned: LOG.debug('pinning "%s": already pinned', pkg_path) return ipkg ipkg.status.is_pinned = True self._write_manifest() LOG.debug('pinned "%s"', pkg_path) return ipkg def unpin(self, pkg_path): """Unpin a currently installed package and allow it to be upgraded. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: :class:`.package.InstalledPackage`: None if no matching installed package could be found, else the installed package that was unpinned. Raises: IOError: when the manifest file can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('unpinning "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('unpinning "%s": no matching package', pkg_path) return None if not ipkg.status.is_pinned: LOG.debug('unpinning "%s": already unpinned', pkg_path) return ipkg ipkg.status.is_pinned = False self._write_manifest() LOG.debug('unpinned "%s"', pkg_path) return ipkg def load(self, pkg_path): """Mark an installed package as being "loaded". The collection of "loaded" packages is a convenient way for Zeek to more simply load a whole group of packages installed via the package manager. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: str: empty string if the package is successfully marked as loaded, else an explanation of why it failed. Raises: IOError: if the loader script or manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('loading "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('loading "%s": no matching package', pkg_path) return "no such package" if ipkg.status.is_loaded: LOG.debug('loading "%s": already loaded', pkg_path) return "" pkg_load_script = os.path.join( self.script_dir, ipkg.package.name, "__load__.zeek" ) if not os.path.exists(pkg_load_script) and not self.has_plugin(ipkg): LOG.debug( 'loading "%s": %s not found and package has no plugin', pkg_path, pkg_load_script, ) return "no __load__.zeek within package script_dir and no plugin included" ipkg.status.is_loaded = True self._write_autoloader() self._write_manifest() self._write_plugin_magic(ipkg) LOG.debug('loaded "%s"', pkg_path) return "" def loaded_package_states(self): """Save "loaded" state for all installed packages. Returns: dict: dictionary of "loaded" status for installed packages """ return { name: ipkg.status.is_loaded for name, ipkg in self.installed_pkgs.items() } def restore_loaded_package_states(self, saved_state): """Restores state for installed packages. Args: saved_state (dict): dictionary of saved "loaded" state for installed packages. """ for pkg_name, ipkg in self.installed_pkgs.items(): if ipkg.status.is_loaded == saved_state[pkg_name]: continue ipkg.status.is_loaded = saved_state[pkg_name] self._write_plugin_magic(ipkg) self._write_autoloader() self._write_manifest() def load_with_dependencies(self, pkg_name, visited=set()): """Mark dependent (but previously installed) packages as being "loaded". Args: pkg_name (str): name of the package. visited (set(str)): set of packages visited along the recursive loading Returns: list(str, str): list of tuples containing dependent package name and whether it was marked as loaded or else an explanation of why the loading failed. """ ipkg = self.find_installed_package(pkg_name) # skip loading a package if it is not installed. if not ipkg: return [(pkg_name, "Loading dependency failed. Package not installed.")] load_error = self.load(pkg_name) if load_error: return [(pkg_name, load_error)] retval = [] visited.add(pkg_name) for pkg in self.get_installed_package_dependencies(pkg_name): if _is_reserved_pkg_name(pkg): continue if pkg in visited: continue retval += self.load_with_dependencies(pkg, visited) return retval def list_depender_pkgs(self, pkg_path): """List of depender packages. If C depends on B and B depends on A, we represent the dependency chain as C -> B -> A. Thus, package C is dependent on A and B, while package B is dependent on just A. Example representation:: { 'A': set(), 'B': set([A, version_of_A]) 'C': set([B, version_of_B]) } Further, package A is a direct dependee for B (and implicitly for C), while B is a direct depender (and C is an implicit depender) for A. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: list: list of depender packages. """ depender_packages, pkg_name = set(), name_from_path(pkg_path) queue = deque([pkg_name]) pkg_dependencies = self.installed_package_dependencies() while queue: item = queue.popleft() for _pkg_name in pkg_dependencies: pkg_dependees = {_pkg for _pkg in pkg_dependencies.get(_pkg_name)} if item in pkg_dependees: # check if there is a cyclic dependency if _pkg_name == pkg_name: return sorted([pkg for pkg in depender_packages] + [pkg_name]) queue.append(_pkg_name) depender_packages.add(_pkg_name) return sorted([pkg for pkg in depender_packages]) def unload_with_unused_dependers(self, pkg_name): """Unmark dependent (but previously installed packages) as being "loaded". Args: pkg_name (str): name of the package. Returns: list(str, str): list of tuples containing dependent package name and whether it was marked as unloaded or else an explanation of why the unloading failed. Raises: IOError: if the loader script or manifest can't be written """ def _has_all_dependers_unloaded(item, dependers): for depender in dependers: ipkg = self.find_installed_package(depender) if ipkg and ipkg.status.is_loaded: return False return True errors = [] queue = deque([pkg_name]) while queue: item = queue.popleft() deps = self.get_installed_package_dependencies(item) for pkg in deps: if _is_reserved_pkg_name(pkg): continue ipkg = self.find_installed_package(pkg) # it is possible that this dependency has been removed via zkg if not ipkg: errors.append((pkg, "Package not installed.")) return errors if ipkg.status.is_loaded: queue.append(pkg) ipkg = self.find_installed_package(item) # it is possible that this package has been removed via zkg if not ipkg: errors.append((item, "Package not installed.")) return errors if ipkg.status.is_loaded: dep_packages = self.list_depender_pkgs(item) # check if there is a cyclic dependency if item in dep_packages: for dep in dep_packages: if item != dep: ipkg = self.find_installed_package(dep) if ipkg and ipkg.status.is_loaded: self.unload(dep) errors.append((dep, "")) self.unload(item) errors.append((item, "")) continue # check if all dependers are unloaded elif _has_all_dependers_unloaded(item, dep_packages): self.unload(item) errors.append((item, "")) continue # package is in use else: dep_packages = self.list_depender_pkgs(pkg_name) dep_listing = "" for _name in dep_packages: dep_listing += f'"{_name}", ' errors.append( ( item, "Package is in use by other packages --- {}.".format( dep_listing[:-2] ), ) ) return errors return errors def unload(self, pkg_path): """Unmark an installed package as being "loaded". The collection of "loaded" packages is a convenient way for Zeek to more simply load a whole group of packages installed via the package manager. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". Returns: bool: True if a package is successfully unmarked as loaded. Raises: IOError: if the loader script or manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('unloading "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if not ipkg: LOG.info('unloading "%s": no matching package', pkg_path) return False if not ipkg.status.is_loaded: LOG.debug('unloading "%s": already unloaded', pkg_path) return True ipkg.status.is_loaded = False self._write_autoloader() self._write_manifest() self._write_plugin_magic(ipkg) LOG.debug('unloaded "%s"', pkg_path) return True def bundle_info(self, bundle_file): """Retrieves information on all packages contained in a bundle. Args: bundle_file (str): the path to the bundle to inspect. Returns: (str, list of (str, str, :class:`.package.PackageInfo`)): a tuple with the the first element set to an empty string if the information successfully retrieved, else an error message explaining why the bundle file was invalid. The second element of the tuple is a list containing information on each package contained in the bundle: the exact git URL and version string from the bundle's manifest along with the package info object retrieved by inspecting git repo contained in the bundle. """ LOG.debug('getting bundle info for file "%s"', bundle_file) bundle_dir = os.path.join(self.scratch_dir, "bundle") delete_path(bundle_dir) make_dir(bundle_dir) infos = [] try: safe_tarfile_extractall(bundle_file, bundle_dir) except Exception as error: return (str(error), infos) manifest_file = os.path.join(bundle_dir, "manifest.txt") config = configparser.ConfigParser(delimiters="=") config.optionxform = str if not config.read(manifest_file): return ("invalid bundle: no manifest file", infos) if not config.has_section("bundle"): return ("invalid bundle: no [bundle] section in manifest file", infos) manifest = config.items("bundle") for git_url, version in manifest: package = Package( git_url=git_url, name=git_url.split("/")[-1], canonical=True ) pkg_path = os.path.join(bundle_dir, package.name) LOG.debug('getting info for bundled package "%s"', package.name) pkg_info = self.info(pkg_path, version=version, prefer_installed=False) infos.append((git_url, version, pkg_info)) return ("", infos) def info(self, pkg_path, version="", prefer_installed=True): """Retrieves information about a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". version (str): may be a git version tag, branch name, or commit hash from which metadata will be pulled. If an empty string is given, then the latest git version tag is used (or the default branch like "main" or "master" if no version tags exist). prefer_installed (bool): if this is set, then the information from any current installation of the package is returned instead of retrieving the latest information from the package's git repo. The `version` parameter is also ignored when this is set as it uses whatever version of the package is currently installed. Returns: A :class:`.package.PackageInfo` object. """ pkg_path = canonical_url(pkg_path) name = name_from_path(pkg_path) if not is_valid_package_name(name): reason = f"Package name {name!r} is not valid." return PackageInfo(Package(git_url=pkg_path), invalid_reason=reason) LOG.debug('getting info on "%s"', pkg_path) # Handle built-in packages like installed packages # but avoid looking up the repository information. bpkg_info = self.find_builtin_package(pkg_path) if prefer_installed and bpkg_info: return bpkg_info ipkg = self.find_installed_package(pkg_path) if prefer_installed and ipkg: status = ipkg.status pkg_name = ipkg.package.name clonepath = os.path.join(self.package_clonedir, pkg_name) clone = git.Repo(clonepath) return _info_from_clone(clone, ipkg.package, status, status.current_version) else: status = None matches = self.match_source_packages(pkg_path) if not matches: package = Package(git_url=pkg_path) try: return self._info(package, status, version) except git.exc.GitCommandError as error: LOG.info( 'getting info on "%s": invalid git repo path: %s', pkg_path, error ) LOG.info('getting info on "%s": matched no source package', pkg_path) reason = ( "package name not found in sources and also" " not a usable git URL (invalid or inaccessible," " use -vvv for details)" ) return PackageInfo(package=package, invalid_reason=reason, status=status) if len(matches) > 1: matches_string = [match.qualified_name() for match in matches] LOG.info( 'getting info on "%s": matched multiple packages: %s', pkg_path, matches_string, ) reason = str.format( '"{}" matches multiple packages, try a more' " specific name from: {}", pkg_path, matches_string, ) return PackageInfo(invalid_reason=reason, status=status) package = matches[0] try: return self._info(package, status, version) except git.exc.GitCommandError as error: LOG.info('getting info on "%s": invalid git repo path: %s', pkg_path, error) reason = "git repository is either invalid or unreachable" return PackageInfo(package=package, invalid_reason=reason, status=status) def _info(self, package, status, version): """Retrieves information about a package. Returns: A :class:`.package.PackageInfo` object. Raises: git.exc.GitCommandError: when failing to clone the package repo """ clonepath = os.path.join(self.scratch_dir, package.name) clone = _clone_package(package, clonepath, version) versions = git_version_tags(clone) if not version: if len(versions): version = versions[-1] else: version = git_default_branch(clone) try: git_checkout(clone, version) except git.exc.GitCommandError: reason = f'no such commit, branch, or version tag: "{version}"' return PackageInfo(package=package, status=status, invalid_reason=reason) LOG.debug('checked out "%s", branch/version "%s"', package, version) return _info_from_clone(clone, package, status, version) def package_versions(self, installed_package): """Returns a list of version number tags available for a package. Args: installed_package (:class:`.package.InstalledPackage`): the package for which version number tags will be retrieved. Returns: list of str: the version number tags. """ name = installed_package.package.name clonepath = os.path.join(self.package_clonedir, name) clone = git.Repo(clonepath) return git_version_tags(clone) def validate_dependencies( self, requested_packages, ignore_installed_packages=False, ignore_suggestions=False, use_builtin_packages=True, ): """Validates package dependencies. Args: requested_packages (list of (str, str)): a list of (package name or git URL, version) string tuples validate. If the version string is empty, the latest available version of the package is used. ignore_installed_packages (bool): whether the dependency analysis should consider installed packages as satisfying dependency requirements. ignore_suggestions (bool): whether the dependency analysis should consider installing dependencies that are marked in another package's 'suggests' metadata field. use_builtin_packages (bool): whether package information from builtin packages is used for dependency resolution. Returns: (str, list of (:class:`.package.PackageInfo`, str, bool)): the first element of the tuple is an empty string if dependency graph was successfully validated, else an error string explaining what is invalid. In the case it was validated, the second element is a list of tuples, each representing a package, where: - The first element is a dependency package that would need to be installed in order to satisfy the dependencies of the requested packages. - The second element of tuples in the list is a version string of the associated package that satisfies dependency requirements. - The third element of the tuples in the list is a boolean value indicating whether the package is included in the list because it's merely suggested by another package. The list will not include any packages that are already installed or that are in the `requested_packages` argument. The list is sorted in dependency order: whenever a dependency in turn has dependencies, those are guaranteed to appear in order in the list. This means that reverse iteration of the list guarantees processing of dependencies prior to the depender packages. """ class Node: def __init__(self, name): self.name = name self.info = None self.requested_version = None # (tracking method, version) self.installed_version = None # (tracking method, version) self.dependers = dict() # name -> version, name needs self at version self.dependees = dict() # name -> version, self needs name at version self.is_suggestion = False def __str__(self): return str.format( "{}\n\trequested: {}\n\tinstalled: {}\n\tdependers: {}\n\tsuggestion: {}", self.name, self.requested_version, self.installed_version, self.dependers, self.is_suggestion, ) graph = dict() # Node.name -> Node, nodes store edges requests = [] # List of Node, just for requested packages # 1. Try to make nodes for everything in the dependency graph... # Add nodes for packages that are requested for installation for name, version in requested_packages: info = self.info(name, version=version, prefer_installed=False) if info.invalid_reason: return ( f'invalid package "{name}": {info.invalid_reason}', [], ) node = Node(info.package.qualified_name()) node.info = info method = node.info.version_type node.requested_version = PackageVersion(method, version) graph[node.name] = node requests.append(node) # Recursively add nodes for all dependencies of requested packages, to_process = copy.copy(graph) while to_process: (_, node) = to_process.popitem() dd = node.info.dependencies(field="depends") ds = node.info.dependencies(field="suggests") if dd is None: return ( str.format('package "{}" has malformed "depends" field', node.name), [], ) all_deps = dd.copy() if not ignore_suggestions: if ds is None: return ( str.format( 'package "{}" has malformed "suggests" field', node.name ), [], ) all_deps.update(ds) for dep_name, _ in all_deps.items(): if dep_name == "zeek": # A zeek node will get added later. continue if dep_name == "zkg": # A zkg node will get added later. continue # Suggestion status propagates to 'depends' field of suggested packages. is_suggestion = ( node.is_suggestion or dep_name in ds and dep_name not in dd ) # If a dependency can be fulfilled by a built-in package # use its PackageInfo directly instead of going through # self.info() to search for it in package sources, where # it may not actually exist. info = None if use_builtin_packages: info = self.find_builtin_package(dep_name) if info is None: info = self.info(dep_name, prefer_installed=False) if info.invalid_reason: return ( str.format( 'package "{}" has invalid dependency "{}": {}', node.name, dep_name, info.invalid_reason, ), [], ) dep_name_orig = dep_name dep_name = info.package.qualified_name() LOG.debug( 'dependency "%s" of "%s" resolved to "%s"', dep_name_orig, node.name, dep_name, ) if dep_name in graph: if graph[dep_name].is_suggestion and not is_suggestion: # Suggestion found to be required by another package. graph[dep_name].is_suggestion = False continue if dep_name in to_process: if to_process[dep_name].is_suggestion and not is_suggestion: # Suggestion found to be required by another package. to_process[dep_name].is_suggestion = False continue node = Node(dep_name) node.info = info node.is_suggestion = is_suggestion graph[node.name] = node to_process[node.name] = node # Add nodes for things that are already installed (including zeek) if not ignore_installed_packages: zeek_version = get_zeek_version() if zeek_version: node = Node("zeek") node.installed_version = PackageVersion( TRACKING_METHOD_VERSION, zeek_version ) graph["zeek"] = node else: LOG.warning('could not get zeek version: no "zeek-config" in PATH ?') node = Node("zkg") node.installed_version = PackageVersion( TRACKING_METHOD_VERSION, __version__ ) graph["zkg"] = node for ipkg in self.installed_packages(): name = ipkg.package.qualified_name() status = ipkg.status if name not in graph: info = self.info(name, prefer_installed=True) node = Node(name) node.info = info graph[node.name] = node graph[name].installed_version = PackageVersion( status.tracking_method, status.current_version, ) # 2. Fill in the edges of the graph with dependency information. for name, node in graph.items(): if name == "zeek": continue if name == "zkg": continue dd = node.info.dependencies(field="depends") ds = node.info.dependencies(field="suggests") if dd is None: return ( str.format('package "{}" has malformed "depends" field', node.name), [], ) all_deps = dd.copy() if not ignore_suggestions: if ds is None: return ( str.format( 'package "{}" has malformed "suggests" field', node.name ), [], ) all_deps.update(ds) for dep_name, dep_version in all_deps.items(): if dep_name == "zeek": if "zeek" in graph: graph["zeek"].dependers[name] = dep_version node.dependees["zeek"] = dep_version elif dep_name == "zkg": if "zkg" in graph: graph["zkg"].dependers[name] = dep_version node.dependees["zkg"] = dep_version else: for _, dependency_node in graph.items(): if dependency_node.name == "zeek": continue if dependency_node.name == "zkg": continue if dependency_node.info.package.matches_path(dep_name): dependency_node.dependers[name] = dep_version node.dependees[dependency_node.name] = dep_version break # 3. Try to solve for a connected graph with no edge conflicts. # Traverse graph in breadth-first order, starting from artificial root # with all nodes requested by caller as child nodes. nodes_todo = requests # The resulting list of packages required to satisfy dependencies, # in depender -> dependent (i.e., root -> leaves in dependency tree) # order. new_pkgs = [] while nodes_todo: node = nodes_todo.pop(0) for name in node.dependees: nodes_todo.append(graph[name]) # Avoid cyclic dependencies: ensure we traverse these edges only # once. (The graph may well be a dag, so it's okay to encounter # specific nodes repeatedly.) node.dependees = [] if not node.dependers: if node.installed_version: # We can ignore packages alreaday installed if nothing else # depends on them. continue if node.requested_version: # Only the packges requested by the caller have a requested # version. We skip those too if nothing depends on them. continue # A new package nothing depends on -- odd? new_pkgs.append( (node.info, node.info.best_version(), node.is_suggestion) ) continue if node.requested_version: # Check that requested version doesn't conflict with dependers. for depender_name, version_spec in node.dependers.items(): msg, fullfills = node.requested_version.fullfills(version_spec) if not fullfills: return ( str.format( 'unsatisfiable dependency: requested "{}" ({}),' ' but "{}" requires {} ({})', node.name, node.requested_version.version, depender_name, version_spec, msg, ), new_pkgs, ) elif node.installed_version: # Check that installed version doesn't conflict with dependers. # track_method, required_version = node.installed_version for depender_name, version_spec in node.dependers.items(): msg, fullfills = node.installed_version.fullfills(version_spec) if not fullfills: return ( str.format( 'unsatisfiable dependency: "{}" ({}) is installed,' ' but "{}" requires {} ({})', node.name, node.installed_version.version, depender_name, version_spec, msg, ), new_pkgs, ) else: # Choose best version that satisfies constraints best_version = None need_branch = False need_version = False def no_best_version_string(node): rval = str.format( '"{}" has no version satisfying dependencies:\n', node.name ) for depender_name, version_spec in node.dependers.items(): rval += str.format( '\t"{}" requires: "{}"\n', depender_name, version_spec ) return rval for _, version_spec in node.dependers.items(): if version_spec.startswith("branch="): need_branch = True elif version_spec != "*": need_version = True if need_branch and need_version: return (no_best_version_string(node), new_pkgs) if need_branch: branch_name = None for depender_name, version_spec in node.dependers.items(): if version_spec == "*": continue if not branch_name: branch_name = version_spec[len("branch=") :] continue if branch_name != version_spec[len("branch=") :]: return (no_best_version_string(node), new_pkgs) if branch_name: best_version = branch_name else: best_version = node.info.default_branch elif need_version: for version in node.info.versions[::-1]: normal_version = normalize_version_tag(version) req_semver = semver.Version.coerce(normal_version) satisfied = True for depender_name, version_spec in node.dependers.items(): try: semver_spec = semver.Spec(version_spec) except ValueError: return ( str.format( 'package "{}" has invalid semver spec: {}', depender_name, version_spec, ), new_pkgs, ) if req_semver not in semver_spec: satisfied = False break if satisfied: best_version = version break if not best_version: return (no_best_version_string(node), new_pkgs) else: # Must have been all '*' wildcards or no dependers best_version = node.info.best_version() new_pkgs.append((node.info, best_version, node.is_suggestion)) # Remove duplicate new nodes, preserving their latest (i.e. deepest-in- # tree) occurrences. Traversing the resulting list right-to-left guarantees # that we never visit a node before we've visited all of its dependees. seen_nodes = set() res = [] for it in reversed(new_pkgs): if it[0].package.name in seen_nodes: continue seen_nodes.add(it[0].package.name) res.insert(0, it) return ("", res) def bundle(self, bundle_file, package_list, prefer_existing_clones=False): """Creates a package bundle. Args: bundle_file (str): filesystem path of the zip file to create. package_list (list of (str, str)): a list of (git URL, version) string tuples to put in the bundle. If the version string is empty, the latest available version of the package is used. prefer_existing_clones (bool): if True and the package list contains a package at a version that is already installed, then the existing git clone of that package is put into the bundle instead of cloning from the remote repository. Returns: str: empty string if the bundle is successfully created, else an error string explaining what failed. """ bundle_dir = os.path.join(self.scratch_dir, "bundle") delete_path(bundle_dir) make_dir(bundle_dir) manifest_file = os.path.join(bundle_dir, "manifest.txt") config = configparser.ConfigParser(delimiters="=") config.optionxform = str config.add_section("bundle") # To be placed into the meta section. builtin_packages = [] def match_package_url_and_version(git_url, version): for ipkg in self.installed_packages(): if ipkg.package.git_url != git_url: continue if ipkg.status.current_version != version: continue return ipkg return None for git_url, version in package_list: # Record built-in packages in the bundle's manifest, but # otherwise ignore them silently. if git_url.startswith(BUILTIN_SCHEME): builtin_packages.append((git_url, version)) continue name = name_from_path(git_url) clonepath = os.path.join(bundle_dir, name) config.set("bundle", git_url, version) if prefer_existing_clones: ipkg = match_package_url_and_version(git_url, version) if ipkg: src = os.path.join(self.package_clonedir, ipkg.package.name) shutil.copytree(src, clonepath, symlinks=True) clone = git.Repo(clonepath) clone.git.reset(hard=True) clone.git.clean("-f", "-x", "-d") for modified_config in self.modified_config_files(ipkg): dst = os.path.join(clonepath, modified_config[0]) shutil.copy2(modified_config[1], dst) continue try: git_clone(git_url, clonepath, shallow=(not is_sha1(version))) except git.exc.GitCommandError as error: return f"failed to clone {git_url}: {error}" # Record the built-in packages expected by this bundle (or simply # installed on the source system) in a new [meta] section to aid # debugging. This isn't interpreted, but if unbundle produces # warnings it may proof helpful. if builtin_packages: config.add_section("meta") entries = [] for git_url, version in builtin_packages: entries.append(f"{name_from_path(git_url)}={version}") config.set("meta", "builtin_packages", ",".join(entries)) with open(manifest_file, "w") as f: config.write(f) archive = shutil.make_archive(bundle_dir, "gztar", bundle_dir) delete_path(bundle_file) shutil.move(archive, bundle_file) return "" def unbundle(self, bundle_file): """Installs all packages contained within a bundle. Args: bundle_file (str): the path to the bundle to install. Returns: str: an empty string if the operation was successful, else an error message indicated what went wrong. """ LOG.debug('unbundle "%s"', bundle_file) bundle_dir = os.path.join(self.scratch_dir, "bundle") delete_path(bundle_dir) make_dir(bundle_dir) try: safe_tarfile_extractall(bundle_file, bundle_dir) except Exception as error: return str(error) manifest_file = os.path.join(bundle_dir, "manifest.txt") config = configparser.ConfigParser(delimiters="=") config.optionxform = str if not config.read(manifest_file): return "invalid bundle: no manifest file" if not config.has_section("bundle"): return "invalid bundle: no [bundle] section in manifest file" manifest = config.items("bundle") for git_url, version in manifest: package = Package( git_url=git_url, name=git_url.split("/")[-1], canonical=True ) # Prepare the clonepath with the contents from the bundle. clonepath = os.path.join(self.package_clonedir, package.name) delete_path(clonepath) shutil.move(os.path.join(bundle_dir, package.name), clonepath) LOG.debug('unbundle installing "%s"', package.name) error = self._install(package, version, use_existing_clone=True) if error: return error # For all the packages that we've just unbundled, verify that their # dependencies are fulfilled through installed packages or built-in # packages and log a warning if not. # # Possible reasons are built-in packages on the source system missing # on the destination system or usage of --nodeps when creating the bundle. for git_url, version in manifest: deps = self.get_installed_package_dependencies(git_url) if deps is None: LOG.warning('package "%s" not installed?', git_url) continue for dep, version_spec in deps.items(): ipkg = self.find_installed_package(dep) if ipkg is None: LOG.warning('dependency "%s" of bundled "%s" missing', dep, git_url) continue msg, fullfills = ipkg.fullfills(version_spec) if not fullfills: LOG.warning( 'dependency "%s" (%s) of "%s" not compatible with "%s"', dep, ipkg.status.current_version, git_url, version_spec, ) return "" def test(self, pkg_path, version="", test_dependencies=False): """Test a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". version (str): if not given, then the latest git version tag is used (or if no version tags exist, the default branch like "main" or "master" is used). If given, it may be either a git version tag or a git branch name. test_dependencies (bool): if True, any dependencies required for the given package will also get tested. Off by default, meaning such dependencies will get locally built and staged, but not tested. Returns: (str, bool, str): a tuple containing an error message string, a boolean indicating whether the tests passed, as well as a path to the directory in which the tests were run. In the case where tests failed, the directory can be inspected to figure out what went wrong. In the case where the error message string is not empty, the error message indicates the reason why tests could not be run. Absence of a test_command in the requested package is considered an error. """ pkg_path = canonical_url(pkg_path) LOG.debug('testing "%s"', pkg_path) pkg_info = self.info(pkg_path, version=version, prefer_installed=False) if pkg_info.invalid_reason: return (pkg_info.invalid_reason, "False", "") if "test_command" not in pkg_info.metadata: return ("Package does not specify a test_command", False, "") if not version: version = pkg_info.metadata_version package = pkg_info.package stage = Stage(self, os.path.join(self.package_testdir, package.name)) stage.populate() request = [(package.qualified_name(), version)] invalid_deps, new_pkgs = self.validate_dependencies(request, False) if invalid_deps: return (invalid_deps, False, stage.state_dir) env, err = stage.get_subprocess_env() if env is None: LOG.warning("%s when running tests for %s", err, package.name) return (err, False, stage.state_dir) pkgs = [] pkgs.append((pkg_info, version)) for info, version, _ in new_pkgs: pkgs.append((info, version)) # Clone all packages, checkout right version, and build/install to # staging area. for info, version in reversed(pkgs): LOG.debug( 'preparing "%s" for testing: version %s', info.package.name, version ) clonepath = os.path.join(stage.clone_dir, info.package.name) # After we prepared the stage, the clonepath might exist (as a # symlink to the installed-version package clone) if we're testing # an alternative version of an installed package. Remove the # symlink. if os.path.islink(clonepath): delete_path(clonepath) try: clone = _clone_package(info.package, clonepath, version) except git.exc.GitCommandError as error: LOG.warning("failed to clone git repo: %s", error) return ( f"failed to clone {info.package.git_url}", False, stage.state_dir, ) try: git_checkout(clone, version) except git.exc.GitCommandError as error: LOG.warning("failed to checkout git repo version: %s", error) return ( str.format( "failed to checkout {} of {}", version, info.package.git_url ), False, stage.state_dir, ) fail_msg = self._stage(info.package, version, clone, stage, env) if fail_msg: return (fail_msg, False, self.state_dir) # Finally, run tests (with correct environment set) if test_dependencies: test_pkgs = pkgs else: test_pkgs = [(pkg_info, version)] for info, version in reversed(test_pkgs): LOG.info('testing "%s"', package) # Interpolate the test command: metadata, invalid_reason = self._interpolate_package_metadata( info.metadata, stage ) if invalid_reason: return (invalid_reason, False, stage.state_dir) if "test_command" not in metadata: LOG.info( 'Skipping unit tests for "%s": no test_command in metadata', info.package.qualified_name(), ) continue test_command = metadata["test_command"] cwd = os.path.join(stage.clone_dir, info.package.name) outfile = os.path.join(cwd, "zkg.test_command.stdout") errfile = os.path.join(cwd, "zkg.test_command.stderr") LOG.debug( 'running test_command for %s with cwd="%s", PATH="%s",' ' and ZEEKPATH="%s": %s', info.package.name, cwd, env["PATH"], env["ZEEKPATH"], test_command, ) with open(outfile, "w") as test_stdout, open(errfile, "w") as test_stderr: cmd = subprocess.Popen( test_command, shell=True, cwd=cwd, env=env, stdout=test_stdout, stderr=test_stderr, ) rc = cmd.wait() if rc != 0: return ( f"test_command failed with exit code {rc}", False, stage.state_dir, ) return ("", True, stage.state_dir) def _get_executables(self, metadata): return metadata.get("executables", "").split() def _stage(self, package, version, clone, stage, env=None): """Stage a package. Staging is the act of getting a package ready for use at a particular location in the file system, called a "stage". The stage may be the actual installation folders for the system's Zeek distribution, or one purely internal to zkg's stage management when testing a package. The steps involved in staging include cloning and checking out the package at the desired version, building it if it features a build_command, and installing script & plugin folders inside the requested stage. Args: package (:class:`.package.Package`): the package to stage version (str): the git tag, branch name, or commit hash of the package version to stage clone (:class:`git.Repo`): the on-disk clone of the package's git repository. stage (:class:`Stage`): the staging object describing the disk locations for installation. env (dict of str -> str): an optional environment to pass to the child process executing the package's build_command, if any. If None, the current environment is used. Returns: str: empty string if staging succeeded, otherwise an error string explaining why it failed. """ LOG.debug('staging "%s": version %s', package, version) metadata_file = _pick_metadata_file(clone.working_dir) metadata_parser = configparser.ConfigParser(interpolation=None) invalid_reason = _parse_package_metadata(metadata_parser, metadata_file) if invalid_reason: return invalid_reason metadata = _get_package_metadata(metadata_parser) metadata, invalid_reason = self._interpolate_package_metadata(metadata, stage) if invalid_reason: return invalid_reason build_command = metadata.get("build_command", "") if build_command: LOG.debug( 'building "%s": running build_command: %s', package, build_command ) bufsize = 4096 build = subprocess.Popen( build_command, shell=True, cwd=clone.working_dir, env=env, bufsize=bufsize, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) try: buildlog = self.package_build_log(clone.working_dir) with open(buildlog, "wb") as f: LOG.info( 'installing "%s": writing build log: %s', package, buildlog ) f.write("=== STDERR ===\n".encode(std_encoding(sys.stderr))) while True: data = build.stderr.read(bufsize) if data: f.write(data) else: break f.write("=== STDOUT ===\n".encode(std_encoding(sys.stdout))) while True: data = build.stdout.read(bufsize) if data: f.write(data) else: break except OSError as error: LOG.warning( 'installing "%s": failed to write build log %s %s: %s', package, buildlog, error.errno, error.strerror, ) returncode = build.wait() if returncode != 0: return f"package build_command failed, see log in {buildlog}" pkg_script_dir = metadata.get("script_dir", "") script_dir_src = os.path.join(clone.working_dir, pkg_script_dir) script_dir_dst = os.path.join(stage.script_dir, package.name) if not os.path.exists(script_dir_src): return str.format( "package's 'script_dir' does not exist: {}", pkg_script_dir ) pkgload = os.path.join(script_dir_src, "__load__.zeek") if os.path.isfile(pkgload): try: symlink_path = os.path.join( os.path.dirname(stage.script_dir), package.name ) make_symlink(os.path.join("packages", package.name), symlink_path) for alias in aliases(metadata): symlink_path = os.path.join( os.path.dirname(stage.script_dir), alias ) make_symlink(os.path.join("packages", package.name), symlink_path) except OSError as exception: error = f"could not create symlink at {symlink_path}" error += f": {type(exception).__name__}: {exception}" return error error = _copy_package_dir( package, "script_dir", script_dir_src, script_dir_dst, self.scratch_dir ) if error: return error else: if "script_dir" in metadata: return str.format( "no __load__.zeek file found" " in package's 'script_dir' : {}", pkg_script_dir, ) else: LOG.warning( 'installing "%s": no __load__.zeek in implicit' " script_dir, skipped installing scripts", package, ) pkg_plugin_dir = metadata.get("plugin_dir", "build") plugin_dir_src = os.path.join(clone.working_dir, pkg_plugin_dir) plugin_dir_dst = os.path.join(stage.plugin_dir, package.name) if not os.path.exists(plugin_dir_src): LOG.info( 'installing "%s": package "plugin_dir" does not exist: %s', package, pkg_plugin_dir, ) if pkg_plugin_dir != "build": # It's common for a package to not have build directory for # plugins, so don't error out in that case, just log it. return str.format( "package's 'plugin_dir' does not exist: {}", pkg_plugin_dir ) error = _copy_package_dir( package, "plugin_dir", plugin_dir_src, plugin_dir_dst, self.scratch_dir ) if error: return error # Ensure any listed executables exist as advertised. for p in self._get_executables(metadata): full_path = os.path.join(clone.working_dir, p) if not os.path.isfile(full_path): return str.format("executable '{}' is missing", p) if not os.access(full_path, os.X_OK): return str.format("file '{}' is not executable", p) if stage.bin_dir is not None: make_symlink( full_path, os.path.join(stage.bin_dir, os.path.basename(p)), force=True, ) return "" def install(self, pkg_path, version=""): """Install a package. Args: pkg_path (str): the full git URL of a package or the shortened path/name that refers to it within a package source. E.g. for a package source called "zeek" with package named "foo" in :file:`alice/zkg.index`, the following inputs may refer to the package: "foo", "alice/foo", or "zeek/alice/foo". version (str): if not given, then the latest git version tag is installed (or if no version tags exist, the default branch like "main" or "master" is installed). If given, it may be either a git version tag, a git branch name, or a git commit hash. Returns: str: empty string if package installation succeeded else an error string explaining why it failed. Raises: IOError: if the manifest can't be written """ pkg_path = canonical_url(pkg_path) LOG.debug('installing "%s"', pkg_path) ipkg = self.find_installed_package(pkg_path) if ipkg: conflict = ipkg.package if conflict.qualified_name().endswith(pkg_path): LOG.debug('installing "%s": re-install: %s', pkg_path, conflict) clonepath = os.path.join(self.package_clonedir, conflict.name) _clone_package(conflict, clonepath, version) return self._install(conflict, version) else: LOG.info( 'installing "%s": matched already installed package: %s', pkg_path, conflict, ) return str.format( 'package with name "{}" ({}) is already installed', conflict.name, conflict, ) matches = self.match_source_packages(pkg_path) if not matches: try: package = Package(git_url=pkg_path) return self._install(package, version) except git.exc.GitCommandError as error: LOG.info('installing "%s": invalid git repo path: %s', pkg_path, error) LOG.info('installing "%s": matched no source package', pkg_path) return "package not found in sources and also not a valid git URL" if len(matches) > 1: matches_string = [match.qualified_name() for match in matches] LOG.info( 'installing "%s": matched multiple packages: %s', pkg_path, matches_string, ) return str.format( '"{}" matches multiple packages, try a more' " specific name from: {}", pkg_path, matches_string, ) try: return self._install(matches[0], version) except git.exc.GitCommandError as error: LOG.warning('installing "%s": source package git repo is invalid', pkg_path) return f'failed to clone package "{pkg_path}": {error}' def _validate_alias_conflict(self, pkg, metadata_dict): """Check if there's an alias conflict. If any of the installed packages aliases collide with the package's name or its aliases, return a string describing the issue. Args: package (:class:`.package.Package`): the package to be installed metadata_dict (dict): The metadata for the given package. package.metadata may not be valid yet. Returns: str: empty string on success, else descriptive error message. """ package_names = {} alias_names = {} for ipkg in self.installed_packages(): if ipkg.package == pkg: continue qn = ipkg.package.qualified_name() package_names[ipkg.package.name] = qn for ipkg_alias in ipkg.package.aliases(): alias_names[ipkg_alias] = qn # Is the new package's name the same as an existing alias? if pkg.name in alias_names: qn = alias_names[pkg.name] return f'name "{pkg.name}" conflicts with alias from "{qn}"' # Any of the aliases matching another package's name or another alias? for alias in aliases(metadata_dict): if alias in package_names: qn = package_names[alias] return ( f'alias "{alias}" conflicts with name of installed package "{qn}"' ) if alias in alias_names: qn = alias_names[alias] return ( f'alias "{alias}" conflicts with alias of installed package "{qn}"' ) return "" def _install(self, package, version, use_existing_clone=False): """Install a :class:`.package.Package`. Returns: str: empty string if package installation succeeded else an error string explaining why it failed. Raises: git.exc.GitCommandError: if the git repo is invalid IOError: if the package manifest file can't be written """ clonepath = os.path.join(self.package_clonedir, package.name) ipkg = self.find_installed_package(package.name) if use_existing_clone or ipkg: clone = git.Repo(clonepath) else: clone = _clone_package(package, clonepath, version) status = PackageStatus() status.is_loaded = ipkg.status.is_loaded if ipkg else False status.is_pinned = ipkg.status.is_pinned if ipkg else False version_tags = git_version_tags(clone) if version: if _is_commit_hash(clone, version): status.tracking_method = TRACKING_METHOD_COMMIT elif version in version_tags: status.tracking_method = TRACKING_METHOD_VERSION else: branches = _get_branch_names(clone) if version in branches: status.tracking_method = TRACKING_METHOD_BRANCH else: LOG.info( 'branch "%s" not in available branches: %s', version, branches ) return f'no such branch or version tag: "{version}"' else: if len(version_tags): version = version_tags[-1] status.tracking_method = TRACKING_METHOD_VERSION else: version = git_default_branch(clone) status.tracking_method = TRACKING_METHOD_BRANCH status.current_version = version git_checkout(clone, version) status.current_hash = clone.head.object.hexsha status.is_outdated = _is_clone_outdated(clone, version, status.tracking_method) metadata_file = _pick_metadata_file(clone.working_dir) metadata_parser = configparser.ConfigParser(interpolation=None) invalid_reason = _parse_package_metadata(metadata_parser, metadata_file) if invalid_reason: return invalid_reason raw_metadata = _get_package_metadata(metadata_parser) invalid_reason = self._validate_alias_conflict(package, raw_metadata) if invalid_reason: return invalid_reason # A dummy stage that uses the actual installation folders; # we do not need to populate() it. stage = Stage(self) fail_msg = self._stage(package, version, clone, stage) if fail_msg: return fail_msg if not package.source: # If installing directly from git URL, see if it actually is found # in a package source and fill in those details. for pkg in self.source_packages(): if pkg.git_url == package.git_url: package.source = pkg.source package.directory = pkg.directory package.metadata = pkg.metadata break package.metadata = raw_metadata self.installed_pkgs[package.name] = InstalledPackage(package, status) self._write_manifest() self._refresh_bin_dir(self.bin_dir) LOG.debug('installed "%s"', package) return "" def _interpolate_package_metadata(self, metadata, stage): # This is a bit circular: we need to parse the user variables, if any, # from the metadata before we can substitute them into other package # metadata. requested_user_vars = UserVar.parse_dict(metadata) if requested_user_vars is None: return None, "package has malformed 'user_vars' metadata field" substitutions = { "zeek_dist": self.zeek_dist, "package_base": stage.clone_dir, } substitutions.update(self.user_vars) for uvar in requested_user_vars: val_from_env = os.environ.get(uvar.name()) if val_from_env: substitutions[uvar.name()] = val_from_env if uvar.name() not in substitutions: substitutions[uvar.name()] = uvar.val() # Now apply the substitutions via a new config parser: metadata_parser = configparser.ConfigParser(defaults=substitutions) metadata_parser.read_dict({"package": metadata}) return _get_package_metadata(metadata_parser), None # Ensure we have links in bin_dir for all executables coming with any of # the currently installed packages. def _refresh_bin_dir(self, bin_dir, prev_bin_dir=None): for ipkg in self.installed_pkgs.values(): for exec in self._get_executables(ipkg.package.metadata): # Put symlinks in place that are missing in current directory src = os.path.join(self.package_clonedir, ipkg.package.name, exec) dst = os.path.join(bin_dir, os.path.basename(exec)) if ( not os.path.exists(dst) or not os.path.islink(dst) or os.path.realpath(src) != os.path.realpath(dst) ): LOG.debug("creating link %s -> %s", src, dst) make_symlink(src, dst, force=True) else: LOG.debug("link %s is up to date", dst) # Remove all links in bin_dir that are associated with executables # coming with any of the currently installed package. def _clear_bin_dir(self, bin_dir): for ipkg in self.installed_pkgs.values(): for exec in self._get_executables(ipkg.package.metadata): old = os.path.join(bin_dir, os.path.basename(exec)) if os.path.islink(old): try: os.unlink(old) LOG.debug("removed link %s", old) except Exception: LOG.warn("failed to remove link %s", old) def _get_branch_names(clone): rval = [] for ref in clone.references: branch_name = str(ref.name) if not branch_name.startswith("origin/"): continue rval.append(branch_name.split("origin/")[1]) return rval def _is_version_outdated(clone, version): version_tags = git_version_tags(clone) latest = normalize_version_tag(version_tags[-1]) return normalize_version_tag(version) != latest def _is_branch_outdated(clone, branch): it = clone.iter_commits("{0}..origin/{0}".format(branch)) num_commits_behind = sum(1 for c in it) return num_commits_behind > 0 def _is_clone_outdated(clone, ref_name, tracking_method): if tracking_method == TRACKING_METHOD_VERSION: return _is_version_outdated(clone, ref_name) elif tracking_method == TRACKING_METHOD_BRANCH: return _is_branch_outdated(clone, ref_name) elif tracking_method == TRACKING_METHOD_COMMIT: return False else: raise NotImplementedError def _is_commit_hash(clone, text): try: commit = clone.commit(text) return commit.hexsha.startswith(text) except Exception: return False def _copy_package_dir(package, dirname, src, dst, scratch_dir): """Copy a directory from a package to its installation location. Returns: str: empty string if package dir copy succeeded else an error string explaining why it failed. """ if not os.path.exists(src): return "" if os.path.isfile(src) and tarfile.is_tarfile(src): tmp_dir = os.path.join(scratch_dir, "untar") delete_path(tmp_dir) make_dir(tmp_dir) try: safe_tarfile_extractall(src, tmp_dir) except Exception as error: return str(error) ld = os.listdir(tmp_dir) if len(ld) != 1: return f"failed to copy package {dirname}: invalid tarfile" src = os.path.join(tmp_dir, ld[0]) if not os.path.isdir(src): return f"failed to copy package {dirname}: not a dir or tarfile" def ignore(_, files): rval = [] for f in files: if f in {".git", "bro-pkg.meta", "zkg.meta"}: rval.append(f) return rval try: copy_over_path(src, dst, ignore=ignore) except shutil.Error as error: errors = error.args[0] reasons = "" for err in errors: src, dst, msg = err reason = f"failed to copy {dirname}: {src} -> {dst}: {msg}" reasons += "\n" + reason LOG.warning('installing "%s": %s', package, reason) return f"failed to copy package {dirname}: {reasons}" return "" def _create_readme(file_path): if os.path.exists(file_path): return with open(file_path, "w") as f: f.write("WARNING: This directory is managed by zkg.\n") f.write("Don't make direct modifications to anything within it.\n") def _clone_package(package, clonepath, version): """Clone a :class:`.package.Package` git repo. Returns: git.Repo: the cloned package Raises: git.exc.GitCommandError: if the git repo is invalid """ delete_path(clonepath) shallow = not is_sha1(version) return git_clone(package.git_url, clonepath, shallow=shallow) def _get_package_metadata(parser): metadata = {item[0]: item[1] for item in parser.items("package")} return metadata def _pick_metadata_file(directory): rval = os.path.join(directory, METADATA_FILENAME) if os.path.exists(rval): return rval return os.path.join(directory, LEGACY_METADATA_FILENAME) def _parse_package_metadata(parser, metadata_file): """Return string explaining why metadata is invalid, or '' if valid.""" if not parser.read(metadata_file): LOG.warning("%s: missing metadata file", metadata_file) return "missing {} (or {}) metadata file".format( METADATA_FILENAME, LEGACY_METADATA_FILENAME ) if not parser.has_section("package"): LOG.warning("%s: metadata missing [package]", metadata_file) return f"{os.path.basename(metadata_file)} is missing [package] section" for a in aliases(_get_package_metadata(parser)): if not is_valid_package_name(a): return f'invalid alias "{a}"' return "" _legacy_metadata_warnings = set() def _info_from_clone(clone, package, status, version): """Retrieves information about a package. Returns: A :class:`.package.PackageInfo` object. """ versions = git_version_tags(clone) default_branch = git_default_branch(clone) if _is_commit_hash(clone, version): version_type = TRACKING_METHOD_COMMIT elif version in versions: version_type = TRACKING_METHOD_VERSION else: version_type = TRACKING_METHOD_BRANCH metadata_file = _pick_metadata_file(clone.working_dir) metadata_parser = configparser.ConfigParser(interpolation=None) invalid_reason = _parse_package_metadata(metadata_parser, metadata_file) if invalid_reason: return PackageInfo( package=package, invalid_reason=invalid_reason, status=status, versions=versions, metadata_version=version, version_type=version_type, metadata_file=metadata_file, default_branch=default_branch, ) if ( os.path.basename(metadata_file) == LEGACY_METADATA_FILENAME and package.qualified_name() not in _legacy_metadata_warnings ): LOG.warning( "Package %s is using the legacy bro-pkg.meta metadata file. " "While bro-pkg.meta still functions, it is recommended to " "use zkg.meta instead for future-proofing. Please report this " "to the package maintainers.", package.qualified_name(), ) _legacy_metadata_warnings.add(package.qualified_name()) metadata = _get_package_metadata(metadata_parser) return PackageInfo( package=package, invalid_reason=invalid_reason, status=status, metadata=metadata, versions=versions, metadata_version=version, version_type=version_type, metadata_file=metadata_file, default_branch=default_branch, ) def _is_reserved_pkg_name(name): return name == "zeek" or name == "zkg" package-manager-3.0.1/zeekpkg/_util.py0000644000175000017500000002577614565163601016002 0ustar rharha""" These are meant to be private utility methods for internal use. """ import errno import importlib.machinery import os import shutil import string import tarfile import types import git import semantic_version as semver def make_dir(path): """Create a directory or do nothing if it already exists. Raises: OSError: if directory cannot be created """ try: os.makedirs(path) except OSError as exception: if exception.errno != errno.EEXIST: raise elif os.path.isfile(path): raise def normalize_version_tag(tag): """Given version string "vX.Y.Z", returns "X.Y.Z". Returns other input strings unchanged. """ if len(tag) > 1 and tag[0] == "v" and tag[1].isdigit(): return tag[1:] return tag def delete_path(path): if os.path.islink(path): os.remove(path) return if not os.path.exists(path): return if os.path.isdir(path): shutil.rmtree(path) else: os.remove(path) def copy_over_path(src, dst, ignore=None): delete_path(dst) shutil.copytree(src, dst, symlinks=True, ignore=ignore) def make_symlink(target_path, link_path, force=True): try: os.symlink(target_path, link_path) except OSError as error: if error.errno == errno.EEXIST and force and os.path.islink(link_path): os.remove(link_path) os.symlink(target_path, link_path) else: raise error def safe_tarfile_extractall(tfile, destdir): """Wrapper to tarfile.extractall(), checking for path traversal. This adds the safeguards the Python docs for tarfile.extractall warn about: Never extract archives from untrusted sources without prior inspection. It is possible that files are created outside of path, e.g. members that have absolute filenames starting with "/" or filenames with two dots "..". Args: tfile (str): the tar file to extract destdir (str): the destination directory into which to place contents Raises: Exception: if the tarfile would extract outside destdir """ def is_within_directory(directory, target): abs_directory = os.path.abspath(directory) abs_target = os.path.abspath(target) prefix = os.path.commonprefix([abs_directory, abs_target]) return prefix == abs_directory with tarfile.open(tfile) as tar: for member in tar.getmembers(): member_path = os.path.join(destdir, member.name) if not is_within_directory(destdir, member_path): raise Exception("attempted path traversal in tarfile") tar.extractall(destdir) def find_sentence_end(s): beg = 0 while True: period_idx = s.find(".", beg) if period_idx == -1: return -1 if period_idx == len(s) - 1: return period_idx next_char = s[period_idx + 1] if next_char.isspace(): return period_idx beg = period_idx + 1 def git_clone(git_url, dst_path, shallow=False): if shallow: try: git.Git().clone( git_url, dst_path, "--no-single-branch", recursive=True, depth=1 ) except git.exc.GitCommandError: if not git_url.startswith(".") and not git_url.startswith("/"): # Not a local repo raise if not os.path.exists(os.path.join(git_url, ".git", "shallow")): raise # Some git versions cannot clone from a shallow-clone, so copy # and reset/clean it to a pristine condition. copy_over_path(git_url, dst_path) rval = git.Repo(dst_path) rval.git.reset("--hard") rval.git.clean("-ffdx") else: git.Git().clone(git_url, dst_path, recursive=True) rval = git.Repo(dst_path) # This setting of the "origin" remote will be a no-op in most cases, but # for some reason, when cloning from a local directory, the clone may # inherit the "origin" instead of using the local directory as its new # "origin". This is bad in some cases since we do not want to try # fetching from a remote location (e.g. when unbundling). This # unintended inheritence of "origin" seems to only happen when cloning a # local git repo that has submodules ? rval.git.remote("set-url", "origin", git_url) return rval def git_checkout(clone, version): """Checkout a version of a git repo along with any associated submodules. Args: clone (git.Repo): the git clone on which to operate version (str): the branch, tag, or commit to checkout Raises: git.exc.GitCommandError: if the git repo is invalid """ clone.git.checkout(version) clone.git.submodule("sync", "--recursive") clone.git.submodule("update", "--recursive", "--init") def git_default_branch(repo): """Return default branch of a git repo, like 'main' or 'master'. If the Git repository has a remote named 'origin', the default branch is taken from the value of its HEAD reference (if it has one). If the Git repository has no remote named 'origin' or that remote has no HEAD, the default branch is selected in this order: 'main' if it exists, 'master' if it exists, the currently checked out branch if any, else the current detached commit. Args: repo (git.Repo): the git clone on which to operate """ try: remote = repo.remote("origin") except ValueError: remote = None if remote: # Technically possible that remote has no HEAD, so guard against that. try: head_ref_name = remote.refs.HEAD.ref.name except Exception: head_ref_name = None if head_ref_name: remote_prefix = "origin/" if head_ref_name.startswith(remote_prefix): return head_ref_name[len(remote_prefix) :] return head_ref_name ref_names = [ref.name for ref in repo.refs] if "main" in ref_names: return "main" if "master" in ref_names: return "master" try: # See if there's a branch currently checked out return repo.head.ref.name except TypeError: # No branch checked out, return commit hash return repo.head.object.hexsha def git_version_tags(repo): """Returns semver-sorted list of version tag strings in the given repo.""" tags = [] for tagref in repo.tags: tag = str(tagref.name) normal_tag = normalize_version_tag(tag) try: sv = semver.Version.coerce(normal_tag) except ValueError: # Skip tags that aren't compatible semantic versions. continue else: tags.append((normal_tag, tag, sv)) return [t[1] for t in sorted(tags, key=lambda e: e[2])] def git_pull(repo): """Does a git pull followed up a submodule update. Args: clone (git.Repo): the git clone on which to operate Raises: git.exc.GitCommandError: in case of git trouble """ repo.git.pull() repo.git.submodule("sync", "--recursive") repo.git.submodule("update", "--recursive", "--init") def git_remote_urls(repo): """Returns a map of remote name -> URL string for configured remotes. You'd normally use repo.remotes[n].urls for this, but with old git versions (<2.7, e.g. on CentOS 7 at the time of writing) this triggers a "git remote show" (without -n) that will query the remotes, which can fail test cases. We use the config subsystem to query the URLs directly -- one of the fallback mechanisms in GitPython's Remote.urls() implementation. """ remote_details = repo.git.config("--get-regexp", r"remote\..+\.url") remotes = {} for line in remote_details.split("\n"): try: remote, url = line.split(maxsplit=1) remote = remote.split(".")[1] remotes[remote] = url except (ValueError, IndexError): pass return remotes def is_sha1(s): if not s: return False if len(s) != 40: return False hexdigits = set(string.hexdigits.lower()) return all(c in hexdigits for c in s) def is_exe(path): return os.path.isfile(path) and os.access(path, os.X_OK) def find_program(prog_name): path, _ = os.path.split(prog_name) if path: return prog_name if is_exe(prog_name) else "" for path in os.environ["PATH"].split(os.pathsep): path = os.path.join(path.strip('"'), prog_name) if is_exe(path): return path return "" class ZeekInfo: """ Helper class holding information about a Zeek installation. """ def __init__(self, *, zeek: str): self._zeek = zeek @property def zeek(self) -> str: """Path to zeek executable.""" if not self._zeek: raise LookupError('No "zeek" executable in PATH') return self._zeek _zeek_info = None def get_zeek_info() -> ZeekInfo: global _zeek_info if _zeek_info is None: _zeek_info = ZeekInfo( zeek=find_program("zeek"), ) return _zeek_info def std_encoding(stream): if stream.encoding: return stream.encoding import locale if locale.getdefaultlocale()[1] is None: return "utf-8" return locale.getpreferredencoding() def read_zeek_config_line(stdout): return stdout.readline().strip() def get_zeek_version(): zeek_config = find_program("zeek-config") if not zeek_config: return "" import subprocess cmd = subprocess.Popen( [zeek_config, "--version"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, bufsize=1, universal_newlines=True, ) return read_zeek_config_line(cmd.stdout) def load_source(filename): """Loads given Python script from disk. Args: filename (str): name of a Python script file Returns: types.ModuleType: a module representing the loaded file """ # https://stackoverflow.com/questions/67631/how-to-import-a-module-given-the-full-path # We currrently require Python 3.5+, where the following looks sufficient: absname = os.path.abspath(filename) dirname = os.path.dirname(absname) # Naming here is unimportant, since we access members of the new # module via the returned instance. loader = importlib.machinery.SourceFileLoader("template_" + dirname, absname) mod = types.ModuleType(loader.name) loader.exec_module(mod) return mod def configparser_section_dict(parser, section): """Returns a dict representing a ConfigParser section. Args: parser (configparser.ConfigParser): a ConfigParser instance section (str): the name of a config section Returns: dict: a dict with key/val entries corresponding to the requested section, or an empty dict if the given parser has no such section. """ res = {} if not parser.has_section(section): return {} for key, val in parser.items(section): res[key] = val return res package-manager-3.0.1/zeekpkg/__init__.py0000644000175000017500000000117314565163601016406 0ustar rharha""" This package defines a Python interface for installing, managing, querying, and performing other operations on Zeek Packages and Package Sources. The main entry point is the :class:`Manager ` class. This package provides a logger named ``LOG`` to which logging stream handlers may be added in order to help log/debug applications. """ import logging __version__ = "3.0.1" __all__ = ["manager", "package", "source", "template", "uservar"] LOG = logging.getLogger(__name__) LOG.addHandler(logging.NullHandler()) from .manager import * from .package import * from .source import * from .uservar import * package-manager-3.0.1/zeekpkg/source.py0000644000175000017500000001160314565163601016146 0ustar rharha""" A module containing the definition of a "package source": a git repository containing a collection of :file:`zkg.index` (or legacy :file:`bro-pkg.index`) files. These are simple INI files that can describe many Zeek packages. Each section of the file names a Zeek package along with the git URL where it is located and metadata tags that help classify/describe it. """ import configparser import git import os import shutil from . import LOG from .package import name_from_path, Package from ._util import git_default_branch, git_checkout, git_clone #: The name of package index files. INDEX_FILENAME = "zkg.index" LEGACY_INDEX_FILENAME = "bro-pkg.index" #: The name of the package source file where package metadata gets aggregated. AGGREGATE_DATA_FILE = "aggregate.meta" class Source: """A Zeek package source. This class contains properties of a package source like its name, remote git URL, and local git clone. Attributes: name (str): The name of the source as given by a config file key in it's ``[sources]`` section. git_url (str): The git URL of the package source. clone (git.Repo): The local git clone of the package source. """ def __init__(self, name, clone_path, git_url, version=None): """Create a package source. Raises: git.exc.GitCommandError: if the git repo is invalid OSError: if the git repo is invalid and can't be re-initialized """ git_url = os.path.expanduser(git_url) self.name = name self.git_url = git_url self.clone = None try: self.clone = git.Repo(clone_path) except git.exc.NoSuchPathError: LOG.debug('creating source clone of "%s" at %s', name, clone_path) self.clone = git_clone(git_url, clone_path, shallow=True) except git.exc.InvalidGitRepositoryError: LOG.debug('deleting invalid source clone of "%s" at %s', name, clone_path) shutil.rmtree(clone_path) self.clone = git_clone(git_url, clone_path, shallow=True) else: LOG.debug('found source clone of "%s" at %s', name, clone_path) old_url = self.clone.git.config("--local", "--get", "remote.origin.url") if git_url != old_url: LOG.debug( 'url of source "%s" changed from %s to %s, reclone at %s', name, old_url, git_url, clone_path, ) shutil.rmtree(clone_path) self.clone = git_clone(git_url, clone_path, shallow=True) git_checkout(self.clone, version or git_default_branch(self.clone)) def __str__(self): return self.git_url def __repr__(self): return self.git_url def package_index_files(self): """Return a list of paths to package index files in the source.""" rval = [] visited_dirs = set() for root, dirs, files in os.walk(self.clone.working_dir, followlinks=True): stat = os.stat(root) visited_dirs.add((stat.st_dev, stat.st_ino)) dirs_to_visit_next = [] for d in dirs: stat = os.stat(os.path.join(root, d)) if (stat.st_dev, stat.st_ino) not in visited_dirs: dirs_to_visit_next.append(d) dirs[:] = dirs_to_visit_next try: dirs.remove(".git") except ValueError: pass for filename in files: if filename == INDEX_FILENAME or filename == LEGACY_INDEX_FILENAME: rval.append(os.path.join(root, filename)) return sorted(rval) def packages(self): """Return a list of :class:`.package.Package` in the source.""" rval = [] # Use raw parser so no value interpolation takes place. parser = configparser.RawConfigParser() aggregate_file = os.path.join(self.clone.working_dir, AGGREGATE_DATA_FILE) parser.read(aggregate_file) for index_file in self.package_index_files(): relative_path = index_file[len(self.clone.working_dir) + 1 :] directory = os.path.dirname(relative_path) lines = [] with open(index_file) as f: lines = [line.rstrip("\n") for line in f] for url in lines: pkg_name = name_from_path(url) agg_key = os.path.join(directory, pkg_name) metadata = {} if parser.has_section(agg_key): metadata = {key: value for key, value in parser.items(agg_key)} package = Package( git_url=url, source=self.name, directory=directory, metadata=metadata, ) rval.append(package) return rval package-manager-3.0.1/zeekpkg/package.py0000644000175000017500000004621114565163601016244 0ustar rharha""" A module with various data structures used for interacting with and querying the properties and status of Zeek packages. """ import os import re from functools import total_ordering import semantic_version as semver from .uservar import UserVar from ._util import find_sentence_end, normalize_version_tag #: The name of files used by packages to store their metadata. METADATA_FILENAME = "zkg.meta" LEGACY_METADATA_FILENAME = "bro-pkg.meta" TRACKING_METHOD_VERSION = "version" TRACKING_METHOD_BRANCH = "branch" TRACKING_METHOD_COMMIT = "commit" TRACKING_METHOD_BUILTIN = "builtin" BUILTIN_SOURCE = "zeek-builtin" BUILTIN_SCHEME = "zeek-builtin://" PLUGIN_MAGIC_FILE = "__zeek_plugin__" PLUGIN_MAGIC_FILE_DISABLED = "__zeek_plugin__.disabled" LEGACY_PLUGIN_MAGIC_FILE = "__bro_plugin__" LEGACY_PLUGIN_MAGIC_FILE_DISABLED = "__bro_plugin__.disabled" def name_from_path(path): """Returns the name of a package given a path to its git repository.""" return canonical_url(path).split("/")[-1] def canonical_url(path): """Returns the url of a package given a path to its git repo.""" url = path.rstrip("/") if url.startswith(".") or url.startswith("/"): url = os.path.realpath(url) return url def is_valid_name(name): """Returns True if name is a valid package name, else False.""" if name != name.strip(): # Reject names with leading/trailing whitespace return False # For aliases: Do not allow file separators. if "/" in name: return False # Avoid creating hidden files and directories. if name.startswith("."): return False if name in ("package", "packages"): return False return True def aliases(metadata_dict): """Return a list of package aliases found in metadata's 'aliases' field.""" if "aliases" not in metadata_dict: return [] return re.split(r",\s*|\s+", metadata_dict["aliases"]) def tags(metadata_dict): """Return a list of tag strings found in the metadata's 'tags' field.""" if "tags" not in metadata_dict: return [] return re.split(r",\s*", metadata_dict["tags"]) def short_description(metadata_dict): """Returns the first sentence of the metadata's 'desciption' field.""" if "description" not in metadata_dict: return "" description = metadata_dict["description"] lines = description.split("\n") rval = "" for line in lines: line = line.lstrip() rval += " " period_idx = find_sentence_end(line) if period_idx == -1: rval += line else: rval += line[: period_idx + 1] break return rval.lstrip() def user_vars(metadata_dict): """Returns a list of (str, str, str) from metadata's 'user_vars' field. Each entry in the returned list is a the name of a variable, its value, and its description. If the 'user_vars' field is not present, an empty list is returned. If it is malformed, then None is returned. """ uvars = UserVar.parse_dict(metadata_dict) if uvars is None: return None return [(uvar.name(), uvar.val(), uvar.desc()) for uvar in uvars] def dependencies(metadata_dict, field="depends"): """Returns a dictionary of (str, str) based on metadata's dependency field. The keys indicate the name of a package (shorthand name or full git URL). The names 'zeek' or 'zkg' may also be keys that indicate a dependency on a particular Zeek or zkg version. The values indicate a semantic version requirement. If the dependency field is malformed (e.g. number of keys not equal to number of values), then None is returned. """ if field not in metadata_dict: return dict() rval = dict() depends = metadata_dict[field] parts = depends.split() keys = parts[::2] values = parts[1::2] if len(keys) != len(values): return None for i, k in enumerate(keys): if i < len(values): rval[k] = values[i] return rval class PackageVersion: """ Helper class to compare package versions with version specs. """ def __init__(self, method, version): self.method = method self.version = version self.req_semver = None def fullfills(self, version_spec): """ Whether this package version fullfills the given version_spec. Returns: (some message, bool) """ if version_spec == "*": # anything goes return "", True if self.method == TRACKING_METHOD_COMMIT: return f'tracking method commit not compatible with "{version_spec}"', False elif self.method == TRACKING_METHOD_BRANCH: return "tracking method branch and commit", False if version_spec.startswith("branch="): branch = version_spec[len("branch=") :] if version_spec != self.version: return f'branch "{self.version}" not matching "{branch}"', False return "", True else: # TRACKING_METHOD_BRANCH / TRACKING_METHOD_BUILTIN if version_spec.startswith("branch="): branch = version_spec[len("branch=") :] return ( f"branch {branch} requested, but using method {self.method}", False, ) if self.req_semver is None: normal_version = normalize_version_tag(self.version) self.req_semver = semver.Version.coerce(normal_version) try: semver_spec = semver.Spec(version_spec) except ValueError: return f'invalid semver spec: "{version_spec}"', False else: if self.req_semver in semver_spec: return "", True return f"{self.version} not in {version_spec}", False @total_ordering class InstalledPackage: """An installed package and its current status. Attributes: package (:class:`Package`): the installed package status (:class:`PackageStatus`): the status of the installed package """ def __init__(self, package, status): self.package = package self.status = status def __repr__(self): return f"InstalledPackage(package={self.package!r}, status={self.status!r})" def __hash__(self): return hash(str(self.package)) def __eq__(self, other): return str(self.package) == str(other.package) def __lt__(self, other): return str(self.package) < str(other.package) def is_builtin(self): return self.package.is_builtin() def fullfills(self, version_spec): """ Does the current version fullfill version_spec? """ return PackageVersion( self.status.tracking_method, self.status.current_version ).fullfills(version_spec) class PackageStatus: """The status of an installed package. This class contains properties of a package related to how the package manager will operate on it. Attributes: is_loaded (bool): whether a package is marked as "loaded". is_pinned (bool): whether a package is allowed to be upgraded. is_outdated (bool): whether a newer version of the package exists. tracking_method (str): either "branch", "version", "commit", or "builtin" to indicate (respectively) whether package upgrades should stick to a git branch, use git version tags, do nothing because the package is to always use a specific git commit hash, or do nothing because the package is built into Zeek. current_version (str): the current version of the installed package, which is either a git branch name or a git version tag. current_hash (str): the git sha1 hash associated with installed package's current version/commit. """ def __init__( self, is_loaded=False, is_pinned=False, is_outdated=False, tracking_method=None, current_version=None, current_hash=None, ): self.is_loaded = is_loaded self.is_pinned = is_pinned self.is_outdated = is_outdated self.tracking_method = tracking_method self.current_version = current_version self.current_hash = current_hash def __repr__(self): member_str = ", ".join(f"{k}={v!r}" for (k, v) in self.__dict__.items()) return f"PackageStatus({member_str})" class PackageInfo: """Contains information on an arbitrary package. If the package is installed, then its status is also available. Attributes: package (:class:`Package`): the relevant Zeek package status (:class:`PackageStatus`): this attribute is set for installed packages metadata (dict of str -> str): the contents of the package's :file:`zkg.meta` or :file:`bro-pkg.meta` versions (list of str): a list of the package's availabe git version tags metadata_version: the package version that the metadata is from version_type: either 'version', 'branch', or 'commit' to indicate whether the package info/metadata was taken from a release version tag, a branch, or a specific commit hash. invalid_reason (str): this attribute is set when there is a problem with gathering package information and explains what went wrong. metadata_file: the absolute path to the :file:`zkg.meta` or :file:`bro-pkg.meta` for this package. Use this if you'd like to parse the metadata yourself. May not be defined, in which case the value is None. """ def __init__( self, package=None, status=None, metadata=None, versions=None, metadata_version="", invalid_reason="", version_type="", metadata_file=None, default_branch=None, ): self.package = package self.status = status self.metadata = {} if metadata is None else metadata self.versions = [] if versions is None else versions self.metadata_version = metadata_version self.version_type = version_type self.invalid_reason = invalid_reason self.metadata_file = metadata_file self.default_branch = default_branch def aliases(self): """Return a list of package name aliases. The canonical one is listed first. """ return aliases(self.metadata) def tags(self): """Return a list of keyword tags associated with the package. This will be the contents of the package's `tags` field.""" return tags(self.metadata) def short_description(self): """Return a short description of the package. This will be the first sentence of the package's 'description' field.""" return short_description(self.metadata) def dependencies(self, field="depends"): """Returns a dictionary of dependency -> version strings. The keys indicate the name of a package (shorthand name or full git URL). The names 'zeek' or 'zkg' may also be keys that indicate a dependency on a particular Zeek or zkg version. The values indicate a semantic version requirement. If the dependency field is malformed (e.g. number of keys not equal to number of values), then None is returned. """ return dependencies(self.metadata, field) def user_vars(self): """Returns a list of user variables parsed from metadata's 'user_vars' field. If the 'user_vars' field is not present, an empty list is returned. If it is malformed, then None is returned. Returns: list of zeekpkg.uservar.UserVar, or None on error """ return UserVar.parse_dict(self.metadata) def best_version(self): """Returns the best/latest version of the package that is available. If the package has any git release tags, this returns the highest one, else it returns the default branch like 'main' or 'master'. """ if self.versions: return self.versions[-1] return self.default_branch def is_builtin(self): return self.package and self.package.is_builtin() def __repr__(self): return f"PackageInfo(package={self.package!r}, status={self.status!r})" @total_ordering class Package: """A Zeek package. This class contains properties of a package that are defined by the package git repository itself and the package source it came from. Attributes: git_url (str): the git URL which uniquely identifies where the Zeek package is located name (str): the canonical name of the package, which is always the last component of the git URL path source (str): the package source this package comes from, which may be empty if the package is not a part of a source (i.e. the user is referring directly to the package's git URL). directory (str): the directory within the package source where the :file:`zkg.index` containing this package is located. E.g. if the package source has a package named "foo" declared in :file:`alice/zkg.index`, then `dir` is equal to "alice". It may also be empty if the package is not part of a package source or if it's located in a top-level :file:`zkg.index` file. metadata (dict of str -> str): the contents of the package's :file:`zkg.meta` or :file:`bro-pkg.meta` file. If the package has not been installed then this information may come from the last aggregation of the source's :file:`aggregate.meta` file (it may not be accurate/up-to-date). """ def __init__( self, git_url, source="", directory="", metadata=None, name=None, canonical=False, ): self.git_url = git_url self.source = source self.directory = directory self.metadata = {} if metadata is None else metadata self.name = name if not canonical: self.git_url = canonical_url(git_url) if not source and os.path.exists(git_url): # Ensures getting real path of relative directories. # e.g. canonical_url catches "./foo" but not "foo" self.git_url = os.path.realpath(self.git_url) self.name = name_from_path(git_url) def __str__(self): return self.qualified_name() def __repr__(self): return ( f"Package(git_url={self.git_url!r}, source={self.source!r}," f" directory={self.directory!r} name={self.name!r})" ) def is_builtin(self): return self.git_url.startswith(BUILTIN_SCHEME) def __hash__(self): return hash(str(self)) def __eq__(self, other): return str(self) == str(other) def __lt__(self, other): return str(self) < str(other) def aliases(self): """Return a list of package name aliases. The canonical one is listed first. """ return aliases(self.metadata) def tags(self): """Return a list of keyword tags associated with the package. This will be the contents of the package's `tags` field and may return results from the source's aggregated metadata if the package has not been installed yet.""" return tags(self.metadata) def short_description(self): """Return a short description of the package. This will be the first sentence of the package's 'description' field and may return results from the source's aggregated metadata if the package has not been installed yet.""" return short_description(self.metadata) def dependencies(self, field="depends"): """Returns a dictionary of dependency -> version strings. The keys indicate the name of a package (shorthand name or full git URL). The names 'zeek' or 'zkg' may also be keys that indicate a dependency on a particular Zeek or zkg version. The values indicate a semantic version requirement. If the dependency field is malformed (e.g. number of keys not equal to number of values), then None is returned. """ return dependencies(self.metadata, field) def user_vars(self): """Returns a list of (str, str, str) from metadata's 'user_vars' field. Each entry in the returned list is a the name of a variable, it's value, and its description. If the 'user_vars' field is not present, an empty list is returned. If it is malformed, then None is returned. """ return user_vars(self.metadata) def name_with_source_directory(self): """Return the package's name within its package source. E.g. for a package source with a package named "foo" in :file:`alice/zkg.index`, this method returns "alice/foo". If the package has no source or sub-directory within the source, then just the package name is returned. """ if self.directory: return f"{self.directory}/{self.name}" return self.name def qualified_name(self): """Return the shortest name that qualifies/distinguishes the package. If the package is part of a source, then this returns "source_name/:meth:`name_with_source_directory()`", else the package's git URL is returned. """ if self.source: return f"{self.source}/{self.name_with_source_directory()}" return self.git_url def matches_path(self, path): """Return whether this package has a matching path/name. E.g for a package with :meth:`qualified_name()` of "zeek/alice/foo", the following inputs will match: "foo", "alice/foo", "zeek/alice/foo" """ path_parts = path.split("/") if self.source: pkg_path = self.qualified_name() pkg_path_parts = pkg_path.split("/") for i, part in reversed(list(enumerate(path_parts))): ri = i - len(path_parts) if part != pkg_path_parts[ri]: return False return True else: if len(path_parts) == 1 and path_parts[-1] == self.name: return True return path == self.git_url def make_builtin_package(*, name: str, current_version: str, current_hash: str = None): """ Given ``name``, ``version`` and ``commit`` as found in Zeek's ``zkg.provides`` entry, construct a :class:`PackageInfo` instance representing the built-in package. """ package = Package( git_url=f"{BUILTIN_SCHEME}{name}", name=name, source=BUILTIN_SOURCE, canonical=True, ) status = PackageStatus( is_loaded=True, # May not hold in the future? is_outdated=False, is_pinned=True, tracking_method=TRACKING_METHOD_BUILTIN, current_version=current_version, current_hash=current_hash, ) return PackageInfo( package=package, status=status, versions=[current_version], ) package-manager-3.0.1/zeekpkg/template.py0000644000175000017500000007007614565163601016472 0ustar rharha""" A module for instantiating different types of Zeek packages. """ import abc import configparser import re import os import shutil import semantic_version as semver import git from . import ( __version__, LOG, ) from .package import ( METADATA_FILENAME, name_from_path, ) from ._util import ( delete_path, git_checkout, git_clone, git_default_branch, git_pull, git_version_tags, git_remote_urls, load_source, make_dir, ) API_VERSION = "1.1.0" class Error(Exception): """Base class for any template-related errors.""" class InputError(Error): """Something's amiss in the input arguments for a package.""" class OutputError(Error): """Something's going wrong while producing template output.""" class LoadError(Error): """Something's going wrong while retrieving a template.""" class GitError(LoadError): """There's git trouble while producing template output.""" class Template: """Base class for any template. Templates need to derive from this class in their toplevel __init__.py. Instances of this class pull together the components in a given template and capture their parameterization. """ @staticmethod def load(config, template, version=None): """Template loader. This function instantiates a zeekpkg.template.Template from a template source present either locally on disk or provided as a repo URL. It first uses the template's __init__.py to bootstrap a module on the fly, then instantiates the zeekpkg.template.Template derivative that must be present in it. Args: config (configparser.ConfigParser): a zkg configuration template (str): template source repo, as directory or git URL version (str): if provided, a specific version tag to use. Ignored when "template" is a local directory. Otherwise, the same logic applies as with packages: the most recent version tag is picked, and if no version tags are available, the default branch. Returns: zeekpkg.template.Template derivative Raises: zeekpkg.template.GitError: git hit a problem during cloning/checkout zeekpkg.template.LoadError: the template Python code does not load cleanly """ repo = None if os.path.isdir(template): # We are loading a template from disk. This can be a git # repo, in which case we use it as-is. Version requests do # not apply. This mirrors the behavior for locally cloned # package sources that zkg installs. if version is not None: LOG.warning('ignoring version request "%s" on local template', version) try: repo = git.Repo(template) if not repo.is_dirty(): version = repo.head.ref.commit.hexsha[:8] except git.exc.InvalidGitRepositoryError: pass templatedir = template else: # We're loading from a git URL. We'll maintain it in the # zkg state folder's clone space and support version # requests. template_clonedir = os.path.join( config.get("paths", "state_dir"), "clones", "template" ) templatedir = os.path.join(template_clonedir, name_from_path(template)) make_dir(template_clonedir) try: if os.path.isdir(templatedir): # A repo of the requested name is already cloned locally. repo = git.Repo(templatedir) # If the requested URL is not the (only) remote of # this repo, treat it like a different repo and # clone anew. (It could be a fork of the same # repo, for example, or simply a naming # collision.) cur_remote_urls = set() for remote in repo.remotes: cur_remote_urls |= set(remote.urls) if len(cur_remote_urls) == 1 and template in cur_remote_urls: repo.git.fetch("-f", "--recurse-submodules=yes", "--tags") else: delete_path(templatedir) repo = None if repo is None: repo = git_clone(template, templatedir) except git.exc.GitCommandError as error: msg = f'failed to update template "{template}": {error}' LOG.error(msg) raise GitError(msg) from error if version is None: version_tags = git_version_tags(repo) if len(version_tags): version = version_tags[-1] else: version = git_default_branch(repo) try: git_checkout(repo, version) except git.exc.GitCommandError as error: msg = ( 'failed to checkout branch/version "{}" of template {}: {}'.format( version, template, error ) ) LOG.warn(msg) raise GitError(msg) from error try: # If we're on a branch, pull in latest updates. # Pulling fails when on a tag/commit. Accessing the # following raises a TypeError when we're not on a # branch. _ = repo.active_branch git_pull(repo) except TypeError: pass # Not on a branch, do nothing except git.exc.GitCommandError as error: msg = 'failed to update branch "{}" of template {}: {}'.format( version, template, error ) LOG.warning(msg) raise GitError(msg) from error try: mod = load_source(os.path.join(templatedir, "__init__.py")) except Exception as error: msg = f'failed to load template "{template}": {error}' LOG.exception(msg) raise LoadError(msg) from error if not hasattr(mod, "TEMPLATE_API_VERSION"): msg = "template{} does not indicate its API version".format( " version " + version if version else "" ) LOG.error(msg) raise LoadError(msg) # The above guards against absence of TEMPLATE_API_VERSION, so # appease pylint for the rest of this function while we access # it. # pylint: disable=no-member try: is_compat = Template.is_api_compatible(mod.TEMPLATE_API_VERSION) except ValueError: raise LoadError( f'API version string "{mod.TEMPLATE_API_VERSION}" is invalid' ) if not is_compat: msg = "template{} API version is incompatible with zkg ({} vs {})".format( " version " + version if version else "", mod.TEMPLATE_API_VERSION, API_VERSION, ) LOG.error(msg) raise LoadError(msg) return mod.Template(templatedir, mod.TEMPLATE_API_VERSION, version, repo) @staticmethod def is_api_compatible(tmpl_ver): """Validate template API compatibility. Given a semver string describing the API version for which a template was written, verifies that we are compatible with it according to semantic versioning rules: MAJOR version changes when we make incompatible API changes MINOR version changes when you add backwards-compatible functionality PATCH version changes when you make backwards-compatible bug fixes. Returns: bool indicating whether template is comatible. Raises: ValueError when given version isn't semver-parseable """ tmpl_sv = semver.Version(tmpl_ver) api_sv = semver.Version(API_VERSION) # A different major version is incompatible by definition if tmpl_sv.major != api_sv.major: return False # Minor version of template can be no larger than ours. if tmpl_sv.minor > api_sv.minor: return False # Patch level does not matter. If ours is less than the # template's we're buggy, but the difference doesn't affect API. return True def __init__(self, templatedir, api_version, version=None, repo=None): """Creates a template. Template objects start from a local directory, and potentially have a version specified. They support the definition and lookup of parameters required during the creation of package instances from the template. They derive these parameters from user variables the template provides and that zkg prompts for upon instantiation. Args: templatedir (str): path to template sources on disk api_version (str): API version targeted by the template (via TEMPLATE_API_VERSION string) version (str): version string of this instance (optional) repo (git.Repo): git repo if this template has one (optional) """ self._templatedir = templatedir self._api_version = api_version self._version = version self._repo = repo self._params = {} # str -> str, set via self.define_param() self._user_vars = [] def define_user_vars(self): """Defines the full set of user vars supported by this template. This function defines the complete set of user vars supported by the template content. Instances of zeekpkg.template.Package and zeekpkg.template.Feature declare which of these user vars they require by implementing the needed_user_vars() method, returning the names of those variables. The default implementation declares no user variables. Returns: list of zeekpkg.uservar.Uservar instances """ return [] def apply_user_vars(self, user_vars): """Apply the user variables to this template. Override this by invoking self.define_param() as needed to create template parameters based on the provided user vars. The relationship of user vars to template parameters is up to you. They can be a 1:1 mapping, you can derive additional parameters from a single user var (e.g. to accommodate string suffixes), or you can use a combination of user vars to define a resulting parameter. Args: user_vars (list of zeekpkg.uservar.UserVar): input values for the template. """ def package(self): """Provides a package template to instantiate. If the template provides a Zeek package, return a Package instance from this method. Returns: zeekpkg.template.Package instance """ return None def features(self): # pylint: disable=no-self-use """Provides any additional features templates supported. If the template provides extra features, return each as an instance of zeekpkg.template.Feature instance in a list. By default, a template offers no features. Returns: list of zeekpkg.template.Feature instances """ return [] def templatedir(self): """Returns the path to the template's source tree on disk.""" return self._templatedir def name(self): """A name for this template, derived from the repo URL.""" return name_from_path(self._templatedir) def api_version(self): """The template API version string declared in this instance's module.""" return self._api_version def version(self): """A version string for the template. This can be a git tag, branch, commit hash, or None if we're using the latest version on the default branch. """ return self._version def has_repo(self): """Returns True if this template has a git repository, False otherwise.""" return self._repo is not None def version_branch(self): """Name of the branch the template is on, if any. Returns branch name if this template version is a branch HEAD, None otherwise (i.e. when it's a specific commit or tag, or we have no repository). """ try: # The following raises a TypeError when not on a branch if self._repo and self._repo.active_branch: return self._repo.active_branch.name except TypeError: pass # Not on a branch return None def version_sha(self): """The git commit hash for this template's version. Returns None when this template got instantiated without a git repo, otherwise a string with the full hash value in ASCII. """ try: if self._repo: return self._repo.head.ref.commit.hexsha except Exception: pass return None def define_param(self, name, val): """Defines a parameter of the given name and value.""" self._params[name] = val def lookup_param(self, name, default=""): """Looks up a parameter, falling back to the given default.""" return self._params.get(name, default) def params(self): """Returns current str->str template parameter dict.""" return self._params def info(self): """Returns a dict capturing information about this template This is usable when rendered as JSON, and also serves as the input for our human-readable template information. """ # In the future a template may not provide a full package, # only features overlaid in combination with another template. res = { "api_version": self._api_version, "provides_package": False, } # XXX we should revisit the reported 'origin' value in # API 2.0.0 -- the the ad-hoc strings are less helpful # than simply providing the key only when there's an # actual origin. if self._repo is not None: try: remotes = git_remote_urls(self._repo) res["origin"] = remotes["origin"] except KeyError: res["origin"] = "unavailable" res["versions"] = git_version_tags(self._repo) res["has_repo"] = True else: res["origin"] = "not a git repository" res["versions"] = [] res["has_repo"] = False pkg = self.package() # pylint: disable=assignment-from-none uvars = self.define_user_vars() feature_names = [] res["user_vars"] = {} for uvar in uvars: res["user_vars"][uvar.name()] = { "description": uvar.desc(), "default": uvar.default(), "used_by": [], } if pkg is not None: res["provides_package"] = True for uvar_name in pkg.needed_user_vars(): try: res["user_vars"][uvar_name]["used_by"].append("package") except KeyError: LOG.warning( 'Package requires undefined user var "%s", skipping', uvar_name ) for feature in self.features(): feature_names.append(feature.name()) for uvar_name in feature.needed_user_vars(): try: res["user_vars"][uvar_name]["used_by"].append(feature.name()) except KeyError: LOG.warning( 'Feature "%s" requires undefined user var "%s"', feature.name(), uvar_name, ) res["features"] = sorted(feature_names) return res def _set_user_vars(self, user_vars): """Provides resolved user vars for the template. Used internally.""" self._params = {} self._user_vars = user_vars self.apply_user_vars(user_vars) def _get_user_vars(self): """Accessor to resolved user vars. Used internally.""" return self._user_vars class _Content(metaclass=abc.ABCMeta): """Common functionality for all template content.""" def __init__(self): self._features = [] self._packagedir = None @abc.abstractmethod def contentdir(self): """Subdirectory providing this content in the template tree. Returns: str: relative path to the content directory """ return None def needed_user_vars(self): """Returns a list of user vars names required by this content. Use this function to declare which of the user vars defined by the template (via Template.define_user_vars()) is required by this component. By doing this, the user only needs to input user vars for template components that actually require them. Returns: A list of strings identifying the needed user vars. """ return [] def add_feature(self, feature): self._features.append(feature) def do_validate(self, tmpl): """Main driver for validation of a template's configuration. zkg calls this internally as part of template validation. You'll likely want to focus on validate() for customization. """ self.validate(tmpl) for feature in self._features: feature.validate(tmpl) def validate(self, tmpl): """Validation of template configuration for this component. Override this in your template's code in order to check whether the template parameters (available via tmpl.lookup_param()) are present and correctly formatted. Raise zeekpkg.template.InputError exceptions as needed. Args: tmpl (zeekpkg.template.Template): template context Raises: zeekpkg.template.InputError when failing validation. """ def do_instantiate(self, tmpl, packagedir, use_force=False): """Main driver for instantiating template content. zkg calls this internally as part of template instantiation. You'll likely want to focus on instantiate() for customization. Args: tmpl (zeekpkg.template.Template): template context packagedir (str): output folder for the instantiation use_force (bool): whether to overwrite/recreate files as needed """ self._packagedir = packagedir self.instantiate(tmpl) for feature in self._features: feature.do_instantiate(tmpl, packagedir, use_force=use_force) def instantiate(self, tmpl): """Instantiation of this template component. This substitutes parameters in the template material and instantiates the result in the output directory. Args: tmpl (zeekpkg.template.Template): template context """ for orig_file, path_name, file_name, content in self._walk(tmpl): if os.path.islink(orig_file): self.instantiate_symlink(tmpl, orig_file, path_name, file_name, content) else: self.instantiate_file(tmpl, orig_file, path_name, file_name, content) def instantiate_file(self, tmpl, orig_file, path_name, file_name, content): """Instantiate a regular file in the template. This gets invoked by instantiate() as it traverses the template content. Each invocation sees the content after parameter substitution in file/directory names and file content. Directories get handled implicitly via the path_name of any files contained in it. This implementation writes the content to the output file, overwriting any pre-existing output. Args: tmpl (zeekpkg.template.Template): template context orig_file (str): the absolute input file name, e.g. "/path/to/template/@param@.zeek" path_name (str): the output directory inside the --packagedir file_name (str): the resulting output file name, e.g. "result.zeek" content (str or bytes): the resulting content for the file. """ out_dir = os.path.join(self._packagedir, path_name) out_file = os.path.join(out_dir, file_name) os.makedirs(out_dir, exist_ok=True) try: with open(out_file, "wb") as hdl: hdl.write(content) shutil.copymode(orig_file, out_file) except OSError as error: LOG.warning('I/O error while instantiating "%s": %s', out_file, error) def instantiate_symlink(self, tmpl, orig_file, path_name, file_name, target): """Instantiate a symbolic link in the template. This gets invoked by instantiate() as it traverses the template content. Each invocation sees the symlink target after parameter substitution. Directories get handled implicitly via the path_name of any files contained in it. This implementation deletes existing files and creates the symlink in their place. Args: tmpl (zeekpkg.template.Template): template context orig_file (str): the absolute input file name, e.g. "/path/to/template/@param@.zeek" path_name (str): the output directory inside the --packagedir file_name (str): the resulting output file name, e.g. "result.zeek" target (str): the location the symlink points to. """ out_dir = os.path.join(self._packagedir, path_name) out_file = os.path.join(out_dir, file_name) os.makedirs(out_dir, exist_ok=True) try: delete_path(out_file) os.symlink(target, out_file) except OSError as error: LOG.warning('OS error while creating symlink "%s": %s', out_file, error) def _walk(self, tmpl): """Generator for instantiating template content. This walks over the template source tree, yielding for every file a 4-tuple of the input file name, the output directory, the output file name in that directory, and the file's content. For symlinks, the content is the symlink target, with any applicable parameter subtitutions made. Args: tmpl (zeekpkg.template.Template): template context """ prefix = os.path.join(tmpl.templatedir(), self.contentdir()) for root, _, files in os.walk(prefix): for fname in files: in_file = root + os.sep + fname # Substitute directory and file names out_path = self._replace(tmpl, root[len(prefix) + 1 :]) out_file = self._replace(tmpl, fname) if os.path.islink(in_file): out_content = self._replace(tmpl, os.readlink(in_file)) else: # Substitute file content. try: with open(in_file, "rb") as hdl: out_content = self._replace(tmpl, hdl.read()) except OSError as error: LOG.warning("skipping instantiation of %s: %s", in_file, error) continue yield in_file, out_path, out_file, out_content def _replace(self, tmpl, content): # pylint: disable=no-self-use """Helper for content substitution. Args: tmpl (zeekpkg.template.Template): template context content (str or bytes): unsubstituted template fodder Returns: str or bytes after parameter substitution. """ for name, val in tmpl.params().items(): pat = "@" + name + "@" if not isinstance(content, str): pat = bytes(pat, "ascii") val = bytes(val, "ascii") content = re.sub(pat, val, content, flags=re.IGNORECASE) return content class Package(_Content): """Template content for a Zeek package. This class fills in package-specific functionality but it still abstract. At a minimum, your template's Package derivative needs to implement contentdir(). """ def do_instantiate(self, tmpl, packagedir, use_force=False): self._prepare_packagedir(packagedir) super().do_instantiate(tmpl, packagedir, use_force) self._update_metadata(tmpl) self._git_init(tmpl) def _prepare_packagedir(self, packagedir): os.makedirs(packagedir, exist_ok=True) def _update_metadata(self, tmpl): """Updates the package's zkg.meta with template information. This information allows re-running template instantiation with identical inputs at a later time. """ config = configparser.ConfigParser(delimiters="=") config.optionxform = str manifest_file = os.path.join(self._packagedir, METADATA_FILENAME) # Best-effort: if the template populated the file, adopt the # content, otherwise create with just our metadata. config.read(manifest_file) section = "template" config.remove_section(section) config.add_section(section) config.set(section, "source", tmpl.name()) if tmpl.has_repo(): tmplinfo = tmpl.info() if tmplinfo["origin"] != "unavailable": config.set(section, "source", tmplinfo["origin"]) if tmpl.version(): # If we're on a branch, disambiguate the version by also mentioning # the exact commit. if tmpl.version_branch(): config.set(section, "version", tmpl.version_branch()) config.set(section, "commit", tmpl.version_sha()[:8]) else: config.set(section, "version", tmpl.version()) else: config.set(section, "version", tmpl.version() or "unversioned") config.set(section, "zkg_version", __version__) if self._features: val = ",".join(sorted([f.name() for f in self._features])) config.set(section, "features", val) section = "template_vars" config.remove_section(section) config.add_section(section) for uvar in tmpl._get_user_vars(): # pylint: disable=protected-access if uvar.val() is not None: config.set(section, uvar.name(), uvar.val()) with open(manifest_file, "w") as hdl: config.write(hdl) def _git_init(self, tmpl): """Initialize git repo and commit instantiated content.""" repo = git.Repo.init(self._packagedir) for fname in repo.untracked_files: repo.index.add(fname) features_info = "" if self._features: names = sorted(['"' + f.name() + '"' for f in self._features]) if len(names) == 1: features_info = f", with feature {names[0]}" else: features_info = ", with features " features_info += ", ".join(names[:-1]) features_info += " and " + names[-1] ver_info = tmpl.version() ver_sha = tmpl.version_sha() if ver_info is None: if ver_sha: ver_info = "version " + ver_sha[:8] else: ver_info = "no versioning" else: ver_info = "version " + ver_info if ver_sha: ver_info += " (" + ver_sha[:8] + ")" repo.index.commit( """Initial commit. zkg {} created this package from template "{}" using {}{}.""".format( __version__, tmpl.name(), ver_info, features_info ) ) class Feature(_Content): """Features overlay additional functionality onto a package. This class fills in feature-specific functionality but it still abstract. At a minimum, your template's Feature derivative needs to implement contentdir(). """ def name(self): """A name for this feature. Defaults to its content directory.""" return self.contentdir() or "unnamed" package-manager-3.0.1/.readthedocs.yaml0000644000175000017500000000107114565163601016061 0ustar rharha# .readthedocs.yaml # Read the Docs configuration file # See https://docs.readthedocs.io/en/stable/config-file/v2.html for details # Required version: 2 # Set the version of Python and other tools you might need build: os: ubuntu-22.04 tools: python: "3.11" # Build documentation in the docs/ directory with Sphinx sphinx: configuration: doc/conf.py # We recommend specifying your dependencies to enable reproducible builds: # https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html python: install: - requirements: ./requirements.txt package-manager-3.0.1/CHANGES0000644000175000017500000021026514565163601013634 0ustar rharha3.0.1 | 2024-02-20 10:52:01 -0700 * Update github actions to checkout v3 and upload-artifact v4 (Tim Wojtulewicz, Corelight) 3.0.0-3 | 2024-02-20 10:42:47 -0700 * Update some documentation about bro-pkg.meta (Tim Wojtulewicz, Corelight) * Revert "Remove references to bro-pkg.meta and bro-pkg.index" (Tim Wojtulewicz, Corelight) This reverts commit 964b0e7235febd17e414134737513d600673e77c. 3.0.0 | 2024-02-15 09:21:49 -0500 * Release 3.0.0. 2.13.0-72 | 2024-02-15 09:21:42 -0500 * Revert --ignore-dirty-git addition (Arne Welzel, Corelight) This reverts commit e3de7c2dd70afde8e5512bdb608ec16138aaf525. 2.13.0-70 | 2024-01-24 12:27:36 -0800 * Set language in conf.py to avoid Sphinx warning (Christian Kreibich, Corelight) * Incorporate sphinx-argparse fix for deprecation warnings in regex. (Christian Kreibich, Corelight) 2.13.0-67 | 2024-01-24 09:31:19 -0800 * Add developer docs about zkg's internal use of directories (Christian Kreibich, Corelight) 2.13.0-64 | 2023-11-06 12:49:34 +0100 * manager/package: Use is_valid_name() for alias validation (Arne Welzel, Corelight) Tighten is_valid_name() to disallow filesystem separator and leading dots. * GH-168: manager: Check for alias conflicts (Arne Welzel, Corelight) Prevent aliases to shadow installed packages, or newly installed packages to replaces aliases, or the same aliases from different packages clashing. * _parse_package_metadata: Restrict alias names (Arne Welzel, Corelight) Ensure aliases do not contain overly surprising characters. This is mainly to avoid directory traversals because it can be used to delete existing symlinks. Also, os.path.join("/a/b", "/c") results in "/c", so an alias of "/c" would place a symlink in /. * GH-171: manager: Support __zeek_plugin__ magic file (Arne Welzel, Corelight) Zeek 6.1 switched from __bro_plugin__ to __zeek_plugin__. Modify _write_plugin_magic() to handle the new and the old file. * overview: Replace :note: with note directive (Arne Welzel, Corelight) Looks better and is what we use elsewhere. * doc/conf.py: Use sphinx_rtd_theme also when on_rtd (Arne Welzel, Corelight) Do not rely on it being set implicitly by readthedocs. It's not anymore. * Place .readthedocs.yaml (Arne Welzel, Corelight) * testing: Bump minimum required cmake version rot13 test plugin (Arne Welzel, Corelight) * Fix Github edit links in top-right of HTML pages (Christian Kreibich, Corelight) These links 404'd because they omitted the file suffix, and they didn't really link to anything editable, just the raw file. --- In this case, "suffix" was used which is set by the readthedocs theme in older versions [1]. Seem good to replace with page_source_suffix. [1] https://github.com/readthedocs/sphinx_rtd_theme/pull/1104 2.13.0-51 | 2023-07-17 18:07:42 -0700 * CI: automatically push to PyPI when pushing a git tag (Christian Kreibich, Corelight) * CI: add Github workflow for testing zkg (Christian Kreibich, Corelight) * CI: remove Cirrus setup (Christian Kreibich, Corelight) * Don't analyze __bro_plugin__ file content in tests, only its presence (Christian Kreibich, Corelight) 2.13.0-46 | 2023-06-30 12:47:34 -0700 * Update man page with changes to other documentation (Tim Wojtulewicz, Corelight) * Remove references to bro-pkg.meta and bro-pkg.index (Tim Wojtulewicz, Corelight) * Remove references to legacy bro naming (Tim Wojtulewicz, Corelight) This intentionally leaves behind references to bro-pkg.meta and __bro_plugin__ but removes all of the other references to scripts and files that have been deprecated and/or removed since Zeek 3.0. * User-visible warning when bro-pkg.meta is in use. (Arne Welzel, Corelight) Relates to #119 * bundle/unbundle: Rework built-in dependency check (Arne Welzel, Corelight) Extend unbundle to validate dependencies of contained packages and log a warning if any are not fulfilled or fail to validate. This can happen when a bundle is created with --nodeps or the target system has different built-in packages as compared to the source system. Built-in packages are included in a bundle's manifest.txt file in a new [meta] section. * manager: Do not include built-in packages in manifest (Arne Welzel, Corelight) Bugfix and test for mistakenly recording built-in packages within the manifest.json file. The built-in information is only ever requested from Zeek. * package: Add PackageVersion (Arne Welzel, Corelight) Make checking if a package version fullfills a version_spec re-usable. This certainly changes the messages produced, but doesn't seem there are many tests upset about that. * Add built-in package awareness (Arne Welzel, Corelight) This change uses Zeek 6.0's --build-info flags to discover the "zkg.provides" entry and makes ``zkg`` aware of these as installed packages. It introduces the following changes * A pseudo zeek-builtin:// scheme for git_urls to recognize when a Package instance relates to a built-in package * The package source name "zeek-builtin" is now reserved * A zkg bundle's manifest now contains a new section called `[bundle_builtin]` listing any packages that are expected to exists as built-in packages on the target system. The content is derived from the built-in packages on the system building the bundle. This is somewhat narrow and opinionated, but at the same time avoids the complexity of any constructed scenarios one can come up with here. * A new tracking method "builtin" exists. It should act as version but makes it explicit that a package is built-in. Most commands have been extended to either ignore (purge) or fail (install, remove, test, load, unload, pin, unpin) when they detect a built-in package being operated on. The commands list and info received a ``--include-builtin`` option for explicitly including built-in packages in the output. * zkg: Log warnings by default (Arne Welzel, Corelight) Currently, to see warnings generated by zkg a user has to pass a single -v. Change this behavior by always installing the StreamHandler() with a warning level so that warnings are displayed by default. * package: Add a few __repr__() implementations (Arne Welzel, Corelight) Not having __repr__() implemented makes it more difficult than needed when using print / IPython shell to look around and discover state. Implement some of them to aid that development style. They aren't perfect, but better than not having them at all. * _util: Add ZeekInfo helper class (Arne Welzel, Corelight) Mostly for collecting executable paths for now, but could also see the ZEEKPATH collections being done there down the road. 2.13.0-32 | 2023-06-20 15:55:21 -0700 * Add --ignore-dirty-git option to test and install commands (Tim Wojtulewicz, Corelight) 2.13.0-30 | 2023-04-21 20:37:27 +0200 * Reduce to 2 CPUs per task for CI tasks (Arne Welzel, Corelight) Specifying 8 CPUs and two tasks immediately allocates as many resources as are available for free for a single user on Cirrus. Depending on what else a user is using Cirrus for, execution of zkg tasks may be delayed for significant amounts of time. The test-suite does not run such a long duration that the 4x reduction in CPUs should have a practical impact. * Fix upgrade using test_command of old version (Arne Welzel, Corelight) * GH-137: Change --force to fail on failing tests (Arne Welzel, Corelight) This is a user-visible change in behavior, but to something sane, so hopefully that's acceptable. Currently, `--force` is oftentimes used in scenarios where it should mean "non-interactive". At the same time, `--force` also implies to ignore failing test commands. This is not reasonable behavior, particularly as `--skiptests` is available. This patch changes the behavior and causes install/upgrade to exit with a failure code when tests fail even with `--force` provided. If someone intentionally wants to install a package with broken tests, they can/should use `--skiptests` explicitly instead. That is less surprising and stands out more. Fixes #137 2.13.0-26 | 2023-02-03 09:23:59 +0100 * Fix zkg bundled usage (Arne Welzel, Corelight) This reverts commit 2b11486b09d76df40aafd411aae88cb5cf902850 which naively moved imports above a sys.path.insert()... Adapt flake8 to ignore those import warnings and add a test so hopefully this won't happen again. 2.13.0-24 | 2023-01-17 14:29:45 -0800 * setup.cfg/pre-commit-config: Add flake8 hook and config (Arne Welzel, Corelight) * zkg: Fix comparison with True (Arne Welzel, Corelight) * zkg: Fix unused variables (Arne Welzel, Corelight) * zkg: Imports at top of file (Arne Welzel, Corelight) * manager: Bad logging invocatoins (Arne Welzel, Corelight) * Remove CodeQL (Christian Kreibich, Corelight) 2.13.0-17 | 2023-01-10 12:54:54 -0800 * Replace manual nested loops with set operation. (Benjamin Bannier, Corelight) * Make sure files are properly closed in `setup.py`. (Benjamin Bannier, Corelight) * Automatically update Python sources to use python-3.7 syntax. (Benjamin Bannier, Corelight) * Add `.git-blame-ignore-revs` file. (Benjamin Bannier, Corelight) * Reformat Python code with Black. (Benjamin Bannier, Corelight) * Add GH action to run pre-commit. (Benjamin Bannier, Corelight) * Add pre-commit config. (Benjamin Bannier, Corelight) 2.13.0-9 | 2023-01-05 13:17:20 -0800 * Tweaks to appease CodeQL (Christian Kreibich, Corelight) * Add CodeQL workflow for GitHub code scanning (LGTM Migrator) 2.13.0-5 | 2022-11-03 19:56:59 -0700 * Fix git complaint about unsafe use of file protocol in submodule test (Christian Kreibich, Corelight) 2.13.0-3 | 2022-10-27 16:05:46 -0700 * Add safe wrapper around tarfile.extractall() (Christian Kreibich, Corelight; Charles McFarland, Trellix) * Newline whitespace tweaks for consistency, no actual change (Christian Kreibich, Corelight) 2.13.0 | 2022-06-01 01:18:25 -0700 * Release 2.13.0. 2.12.0-22 | 2022-06-01 01:18:08 -0700 * Add argcomplete support (Christian Kreibich, Corelight) 2.12.0-20 | 2022-04-28 14:12:23 -0700 * Adjust requirements.txt to use Sphinx 3+, and drop Napoleon (Christian Kreibich, Corelight) * Switch to the sphinx-bundled version of the Napoleon add-on (Christian Kreibich, Corelight) 2.12.0-17 | 2022-04-25 10:36:42 -0700 * Add test to verify progress indicator behavior on TTYs (Christian Kreibich, Corelight) * Remove diff-remove-install-ticks canonifier (Christian Kreibich, Corelight) * Add InstallWorker.wait() method and switch progress-dot writers to it (Christian Kreibich, Corelight) 2.12.0-13 | 2021-11-04 16:39:58 -0700 * Request at least Sphinx 2.0 to avoid a dependency problem in RTD (Christian Kreibich, Corelight) 2.12.0-11 | 2021-11-01 11:20:36 -0700 * Expand initializer to set up repos also for our test templates (Christian Kreibich, Corelight) * Improve template source info in fresh zkg.meta (Christian Kreibich, Corelight) The zkg.meta of freshly created packages so far just said "source = ", where derived from the template's storage location (e.g. just "package-template" for the standard template). We now report the origin URL if the template comes with a git repo. * Show precise commit info in template commit when instantiating from branch (Christian Kreibich, Corelight) We previously just showed something like "version = master", which won't be helpful over time. We now incude the exact commit when not installing from a specific commit or tag. * Remove "zeek-config --zeek_dist" requirement from two tests (Christian Kreibich, Corelight) * Modernize cmake requirements in the rot13 plugin-package (Christian Kreibich, Corelight) * Re-enable CI for the feature releases and use new Dockerfile (Christian Kreibich, Corelight) * Rework Dockerfiles to using our binary packages (Christian Kreibich, Corelight) * Add TEST-REQUIRES for zeek-config to three tests that require it (Christian Kreibich, Corelight) 2.12.0 | 2021-10-12 13:58:09 -0700 * Release 2.12.0. 2.11.0-18 | 2021-10-12 13:57:17 -0700 * Support testing package dependencies in Manager.test() (Christian Kreibich, Corelight) * Interpolate test_command (Christian Kreibich, Corelight) * Strengthen the installation-staging concept (Christian Kreibich, Corelight) * Update baseline of test affected by new package install order (Christian Kreibich, Corelight) * Consistently modify the environment during staging and testing (Christian Kreibich, Corelight) * Install/upgrade packages in reverse dependency order (Christian Kreibich, Corelight) * Ensure breadth-first reporting of new nodes in Manager.validate_dependencies() (Christian Kreibich, Corelight) * Install executables during staging (Christian Kreibich, Corelight) 2.11.0-4 | 2021-08-16 14:04:35 -0700 * Accept `uninstall` as an alias for `remove`. (Benjamin Bannier, Corelight) * Incorporate sphinx-argparse upstream fix for aliased commands (Christian Kreibich, Corelight) 2.11.0 | 2021-07-05 20:58:29 -0700 * Release 2.11.0. 2.10.0-10 | 2021-07-05 20:53:39 -0700 * Require zeek-config for two additional tests that will fail without it (Christian Kreibich, Corelight) * Add --fail-on-aggregate-problems to the refresh command (Christian Kreibich, Corelight) When provided (in addition to --aggregate), any package metadata processing problems cause zkg to exit with error. Without the flag, such problems only trigger a warning. * Added resilience when specifying local paths as package git repos (Christian Kreibich, Corelight) In addition to failing when a given path doesn't exist, the git.Repo() constructor also fails when a given path exists but is not a git repository. That scenario caused backtraces with package installs, testing, and bundling when providing such invalid local paths. We now check for this case and provide according output. This adds guards for the install, test, and bundling commands, which are the entry points for providing local URLs. This also expands the test case for invalid repos. * Fix crash in case the user enters just "zkg template" (Christian Kreibich, Corelight) The output of "zkg template" (which will gain various commands over time) now mirrors that when just saying "zkg". Also, the help output in `zkg template --help` no longer double-mentions "template". 2.10.0 | 2021-06-21 19:51:19 -0700 * Release 2.10.0. 2.9.0-10 | 2021-06-21 19:08:22 -0700 * When zkg is bundled with Zeek, prepend Zeek's Python module path (Christian Kreibich, Corelight) Prepending instead of appending ensures that the bundled zkg picks the zeekpkg module shipped with Zeek, not any others available elsewhere in Python's search path. * Improve the package source refresh test (Christian Kreibich, Corelight) This test now also verifies the "refresh --push" behavior. To do this it makes the "remote" package source bare and verifies the log zkg generates at the debug level, with a bit of additional canonification. For better output filtering, the diff-remove-abspath canonifier now recognizes whitespace (it's the same as used by Zeek), and diff-remove-zkg-version now actually filters the current zkg version, not just a related token that appeared in one test. * Add logging to package source refreshes (Christian Kreibich, Corelight) At log level INFO zkg now reports whether the local aggregate.meta was dirty during a "refresh --push" and thus led to a commit; at log level DEBUG it also reports the package additions/changes/drops that end up in such a commit. * Harden "@" interpretation in package source URLs (Christian Kreibich and Arne Welzel, Corelight) urlparse parses scp-style git URLs ("git@github.com:foo/bar") crudely as path-only URLs, exposing us to a risk of confusing user@ prefixes with @version suffixes. We now look for scp-style URLs and transform those explicitly to the "ssh://" schema to disambiguate. 2.9.0-1 | 2021-05-27 13:46:33 -0700 * Update Python requirement to 3.6+ (Jon Siwek, Corelight) Some new code that came in with zkg 2.9.0 does indeed use Python 3.6 features (f-strings), but also early Python versions are generally end-of-life already. 2.9.0 | 2021-05-18 15:07:05 -0700 * Release 2.9.0. 2.8.0-24 | 2021-05-18 15:06:32 -0700 * Add an baseline canonifier to user-mode btest (Jon Siwek, Corelight) * Change templating API_VERSION to 1.0.0 (Jon Siwek, Corelight) 2.8.0-22 | 2021-05-18 13:13:17 -0700 * Minor docstring formatting improvements (Jon Siwek, Corelight) * Add 'uservar' and 'template' modules to API docs (Jon Siwek, Corelight) * Fix sphinxarg extension's use of old Sphinx logging API (Jon Siwek, Corelight) 2.8.0-19 | 2021-05-18 12:03:36 -0700 * Minor developer guide tweaks (Jon Siwek, Corelight) 2.8.0-18 | 2021-04-28 12:21:12 -0700 * Add "template info" command (Christian Kreibich, Corelight) "template info" shows information about the provided template (or the default), in either JSON or plaintext. This verifies whether a given template is loadable and reports supported features, user vars, and tagged versions. * Provide default branch name in our git wrapper (Christian Kreibich, Corelight) Newer git versions trigger a warning when the default branch name is not set, which cluttered up the logs. This adds the default name, preserving "master" since that branch name is currently visible in test baselines. * Add templating functionality to zkg toplevel script (Christian Kreibich, Corelight) - Provide "zkg create" to instantiate a template, with optional features and user vars for parameterization. - ZKG_DEFAULT_TEMPLATE points at the default template repo, https://github.com/zeek/package-template. * Add a "slug" helper for user vars (Christian Kreibich, Corelight) For templates it'll often be handy to render a variable value into a file-system-safe identifier ("slug"). This provides uservar.slugify() for this purpose. * Add Zeek package templating module (Christian Kreibich, Corelight) This provides infrastructure for structuring and instantiating Zeek package templates. Templates are standalone git repos, managed as follows: - Template objects define package templates, including an output directory, a specific package template, any additional features, and parameters substituted when instantiating. Parameters derive from user variables in a way defined by the template. - The Package and Feature classes model template content and allow validation of input parameters and instantiation of content. The package class has additional functionality for initializing new packages as git repos and adding template metadata to its zkg.meta. - A hierarchy of exceptions accommodates template-specific problems. These aren't currently imported into the zeekpkg namespace -- you need to use zeekpkg.template.* when accessing those components. * Add _util.load_source() for sourcing a Python script as a module (Christian Kreibich, Corelight) Templates are driven by a toplevel __init__.py script, which we source via this function. * Prefill the prompt for user vars with current value (Christian Kreibich, Corelight) This preserves the current model of allowing a quick hit of enter to confirm the existing value, but also lets you empty out the value to provide an empty string as the new value. * Refactor user vars into their own class and module (Christian Kreibich, Corelight) - Moves user-var functionality to a new uservar module and adds UserVar, a class representing an individual user variable. These can be created explicitly, parsed from a dict, or parsed from a NAME=VAL string. - Adds support for --user-arg NAME=VAL for the install, unbundle, and upgrade commands. This can be provided repeatedly and lets you provide user variables via the command line. - Slightly tweaks the user prompt to accommodate situations other than package installation. * Migrate some git helpers to _util for broader access (Christian Kreibich, Corelight) 2.8.0 | 2021-03-26 16:50:59 -0700 * Release 2.8.0. 2.7.1-13 | 2021-03-26 16:50:20 -0700 * Check existence of executables only after the build process. (Robin Sommer, Corelight) 2.7.1-11 | 2021-03-24 18:05:00 -0700 * GH-101: include exception output in dependency import error messages (Jon Siwek, Corelight) 2.7.1-9 | 2021-03-24 15:38:03 -0700 * Add support for packages shipping executables. (Robin Sommer, Corelight) This introduces a new "executables" metadata field that a package can set to scripts or binaries that it wants to make available to users. On package installation, zkg will link these executables into a new, central "bin_dir" that users can put into their PATH. By default, that's "/bin", but it can be relocated through a corresponding "bin_dir" entry in the configuration file. On package removal, these symlinks get removed from "bin_dir". Users can automatically update their path through something like PATH=$(zkg config bin_dir):$PATH. 2.7.1-7 | 2021-03-05 16:39:22 -0800 * Update plugin used in test suite for Zeek 4.1 compatibility (Jon Siwek, Corelight) * Update test baselines to include btest header info (Jon Siwek, Corelight) 2.7.1-5 | 2021-03-05 15:56:09 -0800 * Prevent local githooks from interfering with zkg's commands (Christian Kreibich, Corelight) This adds --no-verify during git commits and pushes to prevent githook interference. * Prevent local githooks from interfering with the testsuite (Christian Kreibich, Corelight) This adds a git wrapper script that skips any git configuration done in the user's home, including typical local git hooks. * Update CI to test Zeek 4.0 and not test 3.2 anymore (Jon Siwek, Corelight) 2.7.1 | 2021-02-05 15:29:59 -0800 * Release 2.7.1. 2.7.0-2 | 2021-02-05 15:29:41 -0800 * Teach autoconfig command the --force option (Jon Siwek, Corelight) Allows for skipping confirmation prompts 2.7.0 | 2021-01-25 12:05:08 -0800 * Release 2.7.0. 2.6.1-7 | 2021-01-25 11:54:28 -0800 * Documentation updates for user mode (Christian Kreibich, Corelight) * Expand use of environment variables to override internal settings (Christian Kreibich, Corelight) - ZKG_DEFAULT_SOURCE allows setting an alternative default package source when https://github.com/zeek/packages isn't desirable. An empty value causes no source to be defined. - ZEEK_ZKG_CONFIG_DIR and ZEEK_ZKG_STATE_DIR allow overriding internal storage locations configured during a Zeek-bundled install. This is mainly to aid testing, so the help output does not include these. * Suggest use of --user when zkg detects write-permission problems (Christian Kreibich, Corelight) * Add user mode to zkg (Christian Kreibich, Corelight) The --user flag forces zkg to manage state, including scripts and plugin directories, in ~/.zkg. --user is mutually exclusive to --configfile, and also overrides any ZKG_CONFIG_FILE environment variable. The flag exists for all commands, so you can also use e.g. autoconfig to write out a config file with those settings. As before, "zkg env" reports the environment variables required for operating Zeek with these settings. * Avoid reporting repeated paths in "zkg env" (Christian Kreibich, Corelight) 2.6.1 | 2021-01-06 21:35:39 -0800 * Release 2.6.1. 2.6.0-8 | 2021-01-06 21:33:22 -0800 * Remove unneeded Python module imports (Jon Siwek, Corelight) * Remove excess argument to a string format() (Jon Siwek, Corelight) * Remove an unreachable return statement (Jon Siwek, Corelight) 2.6.0-5 | 2021-01-06 21:02:09 -0800 * Fix missing argument to a function in dependency analysis logic (Jon Siwek, Corelight) 2.6.0-4 | 2021-01-06 20:52:13 -0800 * When installed as a subproject of Zeek, automatically add its bin/ to PATH (Christian Kreibich, Corelight) This helps package installations succeed that require executables from that directory, such as zeek-config. 2.6.0 | 2020-12-12 21:20:21 -0800 * Release 2.6.0. 2.5.0-12 | 2020-12-12 21:18:34 -0800 * Update quickstart docs to note zkg comes installed with Zeek 4.0.0+ (Jon Siwek, Corelight) * Install zkg.1 man page as part of CMake builds (Jon Siwek, Corelight) 2.5.0-10 | 2020-12-12 20:21:32 -0800 * Support cmake-driven installation of zkg when bundled with Zeek (Christian Kreibich, Corelight) This adds cmake-level templating of the toplevel zkg script to substitute paths that let zkg find its own module. When installing independently, this mechanism has no effect, since the zeekpkg module continues to be found via usual PYTHONPATH mechanisms. The templating also lets us adjust the default location of the config and state directories, since in Zeek-bundled installs they are separate. 2.5.0-8 | 2020-12-10 15:25:50 -0800 * Add explanatory error message to zkg for failed imports of dependencies (Jon Siwek, Corelight) If the external gitpython or semantic-version dependencies aren't available when running `zkg`, it now shows an error messages to explain what's required and an example of how to install them. 2.5.0-7 | 2020-12-07 17:19:11 -0800 * Replace remaining use of Thread.isAlive with Thread.is_alive (Christian Kreibich, Corelight) isAlive() has been deprecated for a while. Python 3.9 (Fedora 33's default) no longer includes it, which broke the metadata-depends and metadata-suggests tests. 2.5.0-4 | 2020-12-04 12:43:42 -0800 * Remove deprecated bro-pkg script and bropkg module (Jon Siwek, Corelight) The PyPI bro-pkg package will also no longer be kept in sync with zkg. * Simplify configparser imports/usages (Jon Siwek, Corelight) * Remove future print_function imports (Jon Siwek, Corelight) * Remove configparser from requirements.txt (Jon Siwek, Corelight) 2.5.0 | 2020-12-04 10:30:29 -0800 * Release 2.5.0. 2.4.2-9 | 2020-12-04 10:28:22 -0800 * Extend test suite to cover a package with a "main" default branch (Jon Siwek, Corelight) As opposed to historical default of "master" being the convention. * GH-76: Detect default branch name of packages automatically (Jon Siwek, Corelight) Rather than use the hardcoded 'master' branch as the conventional default, zkg will now attempt to select a package's default branch as follows: If there is a remote named 'origin' and it has a HEAD reference, whatever branch that points to is taken to be the default branch. If there is not a remote named 'origin' or it does not have a HEAD, then the default branch is selected in this order: 'main' if it exists, 'master' if it exists, the currently active branch (HEAD) if there is one, else the commit hash of the current detached head. Note that zkg still prioritized release version tags, like 'v1.0.0', over branches and the default branch selection logic only applies when no such release version tag exists. * Make test cases resilient to non-master default git branch names (Jon Siwek, Corelight) 2.4.2-5 | 2020-11-26 17:38:59 +0000 * Update pip invocations to use explicit `pip3` (Jon Siwek, Corelight) * Remove Python compatibility logic for versions less than 3.5 (Jon Siwek, Corelight) * Update Python invocations to use explicit `python3` (Jon Siwek, Corelight) * Update documentation to reflect new minimum Python 3.5 requirement (Jon Siwek, Corelight) 2.4.2 | 2020-11-10 15:18:54 -0800 * Release 2.4.2. 2.4.1-5 | 2020-11-10 15:17:45 -0800 * Fix/improve dependency resolution failure messages (Jon Siwek, Corelight) The no_best_version_string() function was incorrectly returning a tuple instead of a string. * GH-83: Fix branch-based dependency analysis for packages with no tags (Jon Siwek, Corelight) Branch-based dependency analysis was incorrectly skipped for packages that had no release version tags. * GH-82: Enforce dependency requirements of already-installed packages (Jon Siwek, Corelight) Treat as an error the installing a package that would break dependency requirements of an already-installed package. 2.4.1 | 2020-11-03 14:55:16 -0800 * Release 2.4.1. 2.4.0-13 | 2020-11-03 14:52:09 -0800 * Improve "remove" operation to not unload already-unloaded dependencies (Jon Siwek, Corelight) * Improve how "unload" operation manages dependencies (Jon Siwek, Corelight) * Added a confirmation prompt for the "unload" operation and a `--force` option to bypass it * The confirmation prompt now lists all dependent packages that will also be unloaded as part of the request * Rename the "runtime dependency management" btest (Jon Siwek, Corelight) * GH-78: Improve structure/output of "remove" operations (Jon Siwek, Corelight) * Previously, the remove operation would indicate that all other installed packages besides the one being removed were dependent and needed to be unloaded. No such action would ever happen, it was just incorrect/extraneous output. * Previously, a second "Proceed?" prompt would appear (as part of the logic to ask if it's alright to unload dependers) even if there's only a single, independent package being removed. * With this change, the entire set of depender-packages is gathered up front to display both packages-to-remove and packages-to-unload within the a single "Proceed?" prompt. This makes it clear what the overall results of the remove operation will be right away rather than prompt per package-to-remove asking just-in-time if it's ok to unload dependers. That is, easier for user to make a single decision at the start rather than find out midway that something isn't to their liking and abort then, possibly leaving things in an undesirable state that still needs sorting out. This is also helps streamline the final output of the resulting unload/remove operations which was previously hard to parse since it included redundant information about which were unloaded and also unimportant errors like saying a package couldn't be unloaded at a certain point because it was still in use by another package except that other package was later scheduled to also be unloaded, making that error moot. * Fix various docstring typos/mistakes (Jon Siwek, Corelight) * GH-79: fix dependency-aware (un)loading logic to ignore "reserved" names (Jon Siwek, Corelight) Both `Manager.load_with_dependencies()` and `Manger.unload_with_unused_dependers()` previously could find a package depending on "zeek" (or "zkg") and consider that as the name of a package to (un)load as part of the operation and fail since they're not real packages that are installed. Those operations now simply skip over such reserved names when walking the dependency graph since there's nothing to do for them. 2.4.0-2 | 2020-11-02 13:38:01 -0800 * Remove superfluous "fetch" during git cloning (Jon Siwek, Corelight) Fetching tags used to be necessary for shallow clones, but that now happens implicitly since changing to use `git clone --no-single-branch` (which is also needed for zkg's branch-based version tracking). A secondary `git fetch` after doing a shallow clone also has negative effect of causing errors on older git versions (e.g. v1.8.3.1, currently found in CentOS 7) since they prohibit fetching from a shallow clone. 2.4.0 | 2020-10-20 15:45:32 -0700 * Release 2.4.0. 2.3.1-2 | 2020-10-20 14:26:32 -0700 * GH-74: handle `git checkout` failures during `zkg refresh --aggregate` (Jon Siwek, Corelight) Previously, a checkout failure caused the entire aggregation process to fail, now it will just skip the offending package and later emit a warning that it could not collect its associated metadata. 2.3.1-1 | 2020-10-20 13:37:16 -0700 * GH-75: show more warnings from `zkg refresh --aggregate` (Jon Siwek, Corelight) Package metadata collection issues (e.g. missing metadata file) were not previously surfaced (except when using `zkg -v` verbosity), but these might always be interesting to know about for those performing the aggregation themselves. 2.3.1 | 2020-09-25 12:24:47 -0700 * Release 2.3.1. 2.3.0-5 | 2020-09-25 12:22:22 -0700 * GH-70: When running tests, use any already installed dependencies (Robin Sommer, Corelight) Rather than rebuilding them. * Small tweak to -v to turn any count >= 3 into debug. (Robin Sommer, Corelight) * Simplify installation thread "progress ticks" logic (Jon Siwek, Corelight) And canonify the "runtime" test baseline to not depend on any particular duration required to join() the installing-thread. 2.3.0 | 2020-09-21 16:50:10 -0700 * Release 2.3.0. * Add `package_base` as a pre-defined configuration key for value interpolation. This enables a package to gain access to files from another package in a well-defined manner. For example, if package `foo` wants to access file `xyz.cfg` from package "bar", it can now declate `bar` as a dependency and then use `%(package_base)s/bar/xyz.cfg` from its `build_command`. (Robin Sommer, Corelight) 2.2.1-2 | 2020-08-10 15:41:16 -0700 * Bump CI to use Zeek 3.2.0 (Jon Siwek, Corelight) 2.2.1 | 2020-08-04 18:20:09 -0700 * Release 2.2.1. 2.2.0-2 | 2020-08-04 17:56:08 -0700 * Improve zkg prompt for unloading of dependent packages (Jon Siwek, Corelight) * Prevent a package that has no scripts from autoloading (Jon Siwek, Corelight) If a package has no scripts, explicitly prevent it from being added to the packages.zeek "autoloader" script since that won't ever work. 2.2.0 | 2020-07-07 14:45:41 -0700 * Release 2.2.0. 2.1.2-22 | 2020-07-07 14:44:57 -0700 * Fix Sphinx add_stylesheet deprecation (Jon Siwek, Corelight) * Fix docstring for Manager.list_depender_pkgs() (Jon Siwek, Corelight) 2.1.2-20 | 2020-07-07 14:30:45 -0700 * Add runtime dependency checks for zkg (Mohan Dhawan, Corelight) The "load", "unload" and "remove" operations now all analyze package dependencies and prompt to any load/unload additional packages as needed to satisfy dependencies. 2.1.2-2 | 2020-06-26 12:01:08 -0700 * Fix '"is" with a literal' warning in Python 3.8+ (Jon Siwek, Corelight) 2.1.2-1 | 2020-06-04 11:53:04 -0700 * Update docs for name change of Zeek's aux/ dir (Jon Siwek, Corelight) 2.1.2 | 2020-04-07 17:29:47 -0700 * Release 2.1.2. 2.1.1-3 | 2020-04-07 17:28:27 -0700 * Introduce basic package name validation (Arne Welzel, Corelight) Installing a package called 'packages' doesn't work because of implementation details (and its not sensible anyway). This patch adds some light validation of package names. Besides the reserved names "packages" and "package", it also rejects package names with fronting and trailing whitespace. 2.1.1-1 | 2020-03-20 09:51:50 -0700 * GH-59: add note to quickstart doc about default pip user script dir (Jon Siwek, Corelight) 2.1.1 | 2020-03-20 09:28:39 -0700 * Release 2.1.1. 2.1.0-2 | 2020-03-20 09:26:02 -0700 * GH-60: Fixup urlparse() usage for Python 2 (Arne Welzel, Corelight) (Support for end-of-life Python 2 is still planned to be removed shortly) 2.1.0 | 2020-03-18 15:39:50 -0700 * Release 2.1.0. 2.0.7-12 | 2020-03-18 15:37:52 -0700 * Add --extra-source command line parameter (Arne Welzel, Corelight) Allow to specify sources without changing the config file. The current approach allows overriding sources specified in the configuration file. The option is added to the top-level parser, as a number of commands are using the sources. * Support `@` in source paths to denote branch/revision to use (Arne Welzel, Corelight) Allow an `@` character in the path part of an URL to be used as a specifier which branch/tag/commit to checkout. Shamelessly copied from `pip` where it can be used to specify the revision of a dependency. * Move git_checkout from manager to util (Arne Welzel, Corelight) * Replace remove_trailing_slashes with rstrip('/') (Arne Welzel, Corelight) 2.0.7-5 | 2020-03-16 15:53:08 -0700 * Add Cirrus CI config (Jon Siwek, Corelight) 2.0.7-3 | 2020-02-24 10:12:39 -0800 * Add missing include header to plugin used for tests (Jon Siwek, Corelight) * Do not expect SHELL in environment when running `zkg env` (Arne Welzel) 2.0.7 | 2019-10-14 11:56:18 -0700 * Release 2.0.7. 2.0.6-3 | 2019-10-14 11:53:46 -0700 * GH-55: Add fallback option to getting user_vars. (Vlad Grigorescu) This generally fixes errors when installing packages that use "user_vars", but the user's zkg config file doesn't have a corresponding key. And that can happen simply by (separately) installing multiple packages that each use "user_vars". 2.0.6 | 2019-09-20 13:06:48 -0700 * Release 2.0.6. 2.0.5-2 | 2019-09-20 13:03:11 -0700 * GH-54: improve error message for package test failures (Jon Siwek, Corelight) * Improve debug logging for test_command (Jon Siwek, Corelight) 2.0.5 | 2019-09-18 16:16:21 -0700 * Release 2.0.5. 2.0.4-1 | 2019-09-18 16:09:19 -0700 * Improve dependency handling (Jon Siwek, Corelight) It now won't report unsatisfiable dependencies between two nodes in the dependency graph that are already installed. E.g. previously, if one used the --nodeps flag to install package "foo" that had a failed dependency on Zeek, then installing unrelated package "bar" without --nodeps would emit an error about "foo" dependencies being unsatisfiable even though that has no relation to the current operation. 2.0.4 | 2019-08-28 15:50:13 -0700 * Release 2.0.4. 2.0.3-2 | 2019-08-28 15:49:54 -0700 * Bugfix: incorrect arguments to Manager.has_plugin() (Christian Kreibich, Corelight) 2.0.3 | 2019-08-26 14:18:12 -0700 * Release 2.0.3. 2.0.2-2 | 2019-08-26 14:16:36 -0700 * When loading/unloading a Zeek package, also enable/disable any plugin (Christian Kreibich, Corelight) This patch does this via renaming of the __bro_plugin__ magic file that Zeek looks for when scanning ZEEK_PLUGIN_PATH. To disable a plugin, zkg renames the file to __bro_plugin__.disabled, and renames that file back to __bro_plugin__ when enabling. 2.0.2 | 2019-07-17 10:46:14 -0700 * Release 2.0.2. 2.0.1-2 | 2019-07-17 10:44:50 -0700 * Simplify how "current_hash" is determined for installed packages (Jon Siwek, Corelight) For packages installed via version tag, this will also change the meaning of "current_hash" from being the hexsha of the tag object itself to the hexsha of the commit object pointed to by the tag. 2.0.1 | 2019-07-09 10:37:20 -0700 * Release 2.0.1. 2.0.0-1 | 2019-07-09 10:36:27 -0700 * Fix a broken link in docs (Jon Siwek, Corelight) 2.0.0 | 2019-06-11 19:33:36 -0700 * Release 2.0.0. 1.7.0-30 | 2019-06-11 19:33:13 -0700 * Fix version string replacement via update-changes (Jon Siwek, Corelight) 1.7.0-29 | 2019-06-11 19:26:22 -0700 * Update packaging scripts (Jon Siwek, Corelight) The new name on PyPi will be "zkg", but a duplicate package (with the same version number) under the old "bro-pkg" name is still built and uploaded along with it. * Change name of "bro_dist" Manager ctor arg to "zeek_dist" (Jon Siwek, Corelight) Note this is a breaking API change for those that were creating Manager objects using "bro_dist" as a named parameter. * Add manager.zeekpath and manager.zeek_plugin_path methods (Jon Siwek, Corelight) Same as manager.bropath and manager.bro_plugin_path. * Rename internal util functions that had "bro" in them (Jon Siwek, Corelight) * Replace "Bro" with "Zeek" in all documentation (Jon Siwek, Corelight) * Replace "bro" usages in unit tests (Jon Siwek, Corelight) * Support zkg.index files within package sources (Jon Siwek, Corelight) bro-pkg.index files, if they exist within a package source, are also still accepted and treated in the same way as before. * Allow "zeek" and "zkg" in the "depends" metadata field (Jon Siwek, Corelight) These are equivalent to the "bro" and "bro-pkg" dependencies that already existed. * Add preference for zkg.meta over bro-pkg.meta files (Jon Siwek, Corelight) The new, preferred package metadata file name is zkg.meta, but we'll still accept a bro-pkg.meta if that's all that exists. * Add zeek_dist config option (Jon Siwek, Corelight) This is treated the same way "bro_dist" worked, which is also still used as a fallback if no "zeek_dist" value is set. Package metadata files also now allow either "%(bro_dist)s" or "%(zeek_dist)s" substitutions. * Rename example config file to zkg.config (Jon Siwek, Corelight) * Change default package source name/url from bro to zeek (Jon Siwek, Corelight) * Change default config and state dirs to ~/.zkg (Jon Siwek, Corelight) The old, default ~/.bro-pkg will still be used if it already exists so users will be able to keep their current configurations without having to start over from scratch. * Add support for using zeek-config script (Jon Siwek, Corelight) With fallback to using bro-config if it exists. * Support new Zeek environment variables (Jon Siwek, Corelight) - ZEEKPATH (BROPATH still used as fallback) - ZEEK_PLUGIN_PATH (BRO_PLUGIN_PATH still used as fallback) * Rename bro-pkg man page to zkg (Jon Siwek, Corelight) * Allow ZKG_CONFIG_FILE environment variable (Jon Siwek, Corelight) Same functionality as the old BRO_PKG_CONFIG_FILE (which is still used as a fallback). * Rename internal env. var. used by setup.py (Jon Siwek, Corelight) * Remove gh-pages Makefile target (Jon Siwek, Corelight) * Add zeekpkg module (Jon Siwek, Corelight) The old bropkg module now emits deprecation warnings. * Add zkg executable file (Jon Siwek, Corelight) The bro-pkg script still exists as a wrapper that prints a deprecation warning by default. 1.7.0-7 | 2019-05-03 19:59:57 -0700 * Improve test_command failure output (Jon Siwek, Corelight) 1.7.0-6 | 2019-04-26 10:22:58 -0700 * Fix error when upgrading from older bro-pkg (Jon Siwek, Corelight) 1.7.0-5 | 2019-04-25 18:20:58 -0700 * Fix unbundling packages that contain submodules (Jon Siwek, Corelight) * Add more logging to unbundle process (Jon Siwek, Corelight) 1.7.0-3 | 2019-04-16 11:42:43 -0700 * Support .zeek file extension for upcoming Zeek 3.0 release (Daniel Thayer) 1.7.0 | 2019-04-12 09:27:36 -0700 * Release 1.7.0. 1.6.0-1 | 2019-04-12 09:24:54 -0700 * Add support for packages containing git submodules (Jon Siwek, Corelight) 1.6.0 | 2019-04-03 15:10:00 -0700 * Release 1.6.0. 1.5.6-3 | 2019-04-03 15:09:23 -0700 * Include bro-pkg.meta file info in the "bro-pkg info" output. (Christian Kreibich, Corelight) * Track the full bro-pkg.meta filename in package.PackageInfo. (Christian Kreibich, Corelight) 1.5.6 | 2019-03-27 14:18:39 -0700 * Release 1.5.6. 1.5.5-1 | 2019-03-27 14:16:40 -0700 * GH-47: give more bro-pkg commands a non-zero exit code on fail (Jon Siwek, Corelight) 1.5.5 | 2019-03-27 13:53:59 -0700 * Release 1.5.5. 1.5.4-2 | 2019-03-27 13:52:12 -0700 * GH-47: give `bro-pkg install` non-zero exit code on failure (Jon Siwek, Corelight) * Update documentation regarding --bro-dist (Jon Siwek, Corelight) 1.5.4 | 2019-03-21 20:17:31 -0700 * Release 1.5.4. 1.5.3-1 | 2019-03-21 20:12:41 -0700 * GH-46: fix --version to be able to use commit hash (Jon Siwek, Corelight) 1.5.3 | 2019-03-05 18:46:32 -0800 * Release 1.5.3. 1.5.2-2 | 2019-03-05 18:43:19 -0800 * GH-45: improve compatibility with older git versions (Jon Siwek, Corelight) 1.5.2-1 | 2019-02-13 17:26:48 -0800 * Use .bundle extension in bundle/unbundle docs per convention (Jon Siwek, Corelight) 1.5.2 | 2019-01-11 13:18:51 -0600 * Release 1.5.2. 1.5.1-5 | 2019-01-11 13:18:18 -0600 * Add docs for 'credits' metadata field (Jon Siwek, Corelight) 1.5.1-4 | 2019-01-11 11:59:35 -0600 * Swap Zeek in for Bro in Sphinx config (Jon Siwek, Corelight) 1.5.1-3 | 2019-01-10 17:59:09 -0600 * Fix RTD URL (Jon Siwek, Corelight) 1.5.1-2 | 2019-01-10 17:18:58 -0600 * Update URLs and some simple s/Bro/Zeek in README (Jon Siwek, Corelight) 1.5.1-1 | 2019-01-04 17:17:27 -0600 * GH-41: fix man page formatting of positional args (Jon Siwek, Corelight) 1.5.1 | 2018-12-17 12:21:41 -0600 * Release 1.5.1. * GH-40: improve `bro-pkg bundle --help` output (Jon Siwek, Corelight) 1.5.0 | 2018-12-12 15:32:36 -0600 * Release 1.5.0. 1.4.2-2 | 2018-12-12 15:31:47 -0600 * Add an 'aliases' package metadata field (Jon Siwek, Corelight) This is used for specifying alternate package names that one may @load it by, and may be generally useful when renaming package repositories while still supporting existing package installations under the old name. * Fix bro-pkg to be able to upgrade packages no longer in a source (Jon Siwek, Corelight) 1.4.2 | 2018-08-03 16:46:21 -0500 * Release 1.4.2. 1.4.1-1 | 2018-08-03 16:44:45 -0500 * Fix SafeConfigParser deprecation warning (Jon Siwek, Corelight) 1.4.1 | 2018-07-25 14:37:53 -0500 * Release 1.4.1. 1.4.0-1 | 2018-07-25 14:36:23 -0500 * Improve error message for invalid bundle manifest files (Jon Siwek, Corelight) 1.4.0 | 2018-07-12 11:34:00 -0500 * Release 1.4.0. 1.3.12-7 | 2018-07-12 11:31:14 -0500 * GH-35: allow category argument to `bro-pkg info` (Corelight) 1.3.12-6 | 2018-07-12 10:40:13 -0500 * GH-34: exit non-zero from `bro-pkg info` on lookup failure (Corelight) 1.3.12-5 | 2018-07-12 10:26:35 -0500 * Fix typos (Jon Zeolla) 1.3.12-3 | 2018-07-12 10:15:57 -0500 * Refactor tracking method strings into variables (Corelight) * Support installing packages via commit hash (Corelight) * Stabilize a unit test (Corelight) 1.3.12 | 2018-06-18 09:03:06 -0500 * Release 1.3.12. 1.3.11-3 | 2018-06-18 09:02:39 -0500 * Improve default character encoding handling (Corelight) 1.3.11-2 | 2018-06-18 08:22:17 -0500 * Fix --version for branch names containing a slash '/' (Michael Dopheide) 1.3.11 | 2018-05-29 08:31:19 -0500 * Release 1.3.11. 1.3.10-1 | 2018-05-29 08:29:54 -0500 * Fix invalid Package ctor syntax with Python 2 (Corelight) 1.3.10 | 2018-05-25 14:57:09 -0500 * Release 1.3.10. 1.3.9-1 | 2018-05-25 14:52:08 -0500 * Improve package info canonicalization (Corelight) 1.3.9 | 2018-05-24 16:48:19 -0500 * Fix for packages storing a relative path as their git URL (Corelight) 1.3.8-1 | 2018-05-23 16:00:10 -0500 * Fix typo in README (Corelight) 1.3.8 | 2018-05-10 10:42:00 -0500 * Release 1.3.8. 1.3.7-1 | 2018-05-10 10:39:18 -0500 * GH-32: support version tags prefixed by 'v' (e.g. vX.Y.Z) (Corelight) 1.3.7 | 2018-05-05 08:10:54 -0500 * Release 1.3.7. 1.3.6-1 | 2018-05-05 08:09:45 -0500 * GH-31: Improve error message when failing to find a usable git repo (Corelight) 1.3.6 | 2018-04-25 09:56:45 -0500 * Release 1.3.6. 1.3.5-1 | 2018-04-25 09:56:07 -0500 * GH-30: ignore interpolation errors during metadata aggregation (Corelight) 1.3.5 | 2018-03-28 14:13:29 -0500 * Release 1.3.5. 1.3.4-1 | 2018-03-28 14:12:38 -0500 * GH-29: Move requirements.doc.txt to generalized requirements.txt (Corelight) 1.3.4 | 2018-03-05 09:53:23 -0600 * Release 1.3.4. 1.3.3-2 | 2018-03-05 09:52:56 -0600 * Fix building docs on Sphinx 1.7+ (Corelight) 1.3.3-1 | 2018-03-05 09:31:45 -0600 * Fix running install/test/bundle commands on detached git clones (Corelight) 1.3.3 | 2018-02-15 11:04:09 -0600 * Release 1.3.3. 1.3.2-1 | 2018-02-15 11:02:20 -0600 * GH-26: fix error when config_file is not in script_dir or plugin_dir (Corelight) 1.3.2 | 2018-02-01 17:38:15 -0600 * Release 1.3.2. 1.3.1-5 | 2018-02-01 17:37:34 -0600 * Improve logging stream levels for a couple log messages (Corelight) * Change default confirmation prompt answer to 'no' on test failures (Corelight) 1.3.1-3 | 2018-02-01 17:17:54 -0600 * Minor doc improvements (Corelight) 1.3.1-2 | 2018-02-01 16:13:25 -0600 * Teach bro-pkg to emit an error when operating on dirty git clones When trying to use the install, test, or bundle commands on local git clones that have modifications or untracked files, bro-pkg now aborts with an error message. This is to help catch situations where one thinks their uncommitted local changes will be installed by bro-pkg, but actually will not be unless they are committed in the git repo. (Corelight) 1.3.1-1 | 2018-02-01 15:43:19 -0600 * Fix install command to be able to re-install an existing package Before, using 'install' on a previously-installed package would not correctly update the cached git repository so if there were any changes since the last installation they would not be seen after the operation completed. (Corelight) 1.3.1 | 2017-12-07 19:33:23 -0600 * Release 1.3.1. 1.3-1 | 2017-12-07 19:29:35 -0600 * GH-22: fix 'suggests' field in json output to use key-value pairs (Corelight) 1.3 | 2017-11-28 19:34:15 -0600 * Release 1.3. 1.2.6-2 | 2017-11-28 19:29:38 -0600 * GH-22: add a 'suggests' metadata field This new field can be used in the same way as 'depends', but is used for packages that are merely complementary and not strictly required for the suggesting package to function. A "(suggested)" indicator for install, bundle, and upgrade commands will now appear during their bro-pkg confirmation prompts. These commands also now have a --nosuggestions flag to ignore all suggested packages. This changes the bropkg.manager.validate_dependencies() API to return an additional boolean indicating whether a package was included in the list as a result of suggestions. (Corelight) 1.2.6-1 | 2017-11-27 09:46:02 -0600 * GH-24: improve script_dir metadata field documentation (Corelight) 1.2.6 | 2017-11-23 15:27:46 -0600 * Release 1.2.6. 1.2.5-1 | 2017-11-23 15:27:12 -0600 * GH-23: more "Edit on GitHub" breadcrumb customization. (Corelight) 1.2.5 | 2017-11-23 14:42:33 -0600 * Release 1.2.5. 1.2.4-1 | 2017-11-23 14:41:09 -0600 * GH-23: one more "Edit on GitHub" link fix attempt. (Corelight) 1.2.4 | 2017-11-23 13:57:00 -0600 * Release 1.2.4. 1.2.3-1 | 2017-11-23 13:54:11 -0600 * GH-23: another attempt to remove "Edit on GitHub" link. (Corelight) 1.2.3 | 2017-11-23 13:29:07 -0600 * Release 1.2.3. 1.2.2-2 | 2017-11-23 13:28:15 -0600 * GH-23: remove broken "Edit on GitHub" link from stable version of docs. (Corelight) 1.2.2-1 | 2017-11-23 13:27:06 -0600 * Add a missing dependency for building docs. (Corelight) 1.2.2 | 2017-10-16 16:14:02 -0500 * Release 1.2.2. 1.2.1-1 | 2017-10-16 16:11:37 -0500 * GH-21: again trying to give aggregate metadata consistent ordering (Jon Siwek) 1.2.1 | 2017-10-11 15:03:56 -0500 * Release 1.2.1. 1.2-1 | 2017-10-11 15:02:21 -0500 * Fix building PyPI distribution from Python 3+ It should now still include the configparser backport dependency. (Jon Siwek) 1.2 | 2017-09-29 13:15:37 -0500 * Release 1.2. 1.1-1 | 2017-09-29 13:15:00 -0500 * BIT-1852: allow 'bro-pkg' as a 'depends' field value This allows packages to depend on new features of bro-pkg not available in previous version. (Jon Siwek) 1.1 | 2017-09-28 18:46:30 -0500 * Release 1.1. 1.0.4-32 | 2017-09-28 18:45:54 -0500 * BIT-1852: allow plugin_dir to also accept tarfiles This allows packagers to specify a tarfile containing the Bro plugin instead of a directory. May be useful as the default Bro plugin CMake skeleton will end up producing a tarfile as part of the plugin build process that contains the minimal set of files comprising the plugin. (Jon Siwek) 1.0.4-31 | 2017-09-28 16:16:39 -0500 * GH-9: add new 'user_vars' bro-pkg.meta field and config file section This field allows for packages to solicit users for extra configuration settings which may then be referenced in that package's build_command. Useful for asking users for installation paths of external dependencies, which may differ between users/systems. (Jon Siwek) 1.0.4-30 | 2017-09-27 11:14:52 -0500 * GH-9: show 'external_depends' metadata during 'unbundle' command (Jon Siwek) * Add manager.bundle_info() to API. It retrieves information on all packages in a bundle without installing the bundle. (Jon Siwek) 1.0.4-28 | 2017-09-27 10:14:16 -0500 * Remove an error msg when upgrading packages that have no test_command (Jon Siwek) 1.0.4-27 | 2017-09-26 13:27:53 -0500 * GH-9: allow 'external_depends' field in bro-pkg.meta. The contents of this field get displayed to users in the install and upgrade commands to indicate extra pre-requisite software their OS should have installed. (Jon Siwek) 1.0.4-26 | 2017-09-26 11:41:37 -0500 * Put install's "unit test skipped" msg into log stream instead of stdout. (Jon Siwek) 1.0.4-25 | 2017-09-22 12:26:56 -0500 * GH-15: add --nodeps to install, bundle, and upgrade commands This option allows to bypasses dependency resolution/checking, accepting risk of causing broken/unusable package state. (Jon Siwek) 1.0.4-24 | 2017-09-20 13:11:48 -0500 * Code formatting. (Jon Siwek) 1.0.4-23 | 2017-09-20 13:07:57 -0500 * Add '--allvers' option to output metadata for ALL versions when outputting JSON. (Terry Fleury) * Add JSON output as option to 'info' command. (Terry Fleury) 1.0.4-18 | 2017-09-20 12:52:17 -0500 * Add 'info' option to force read metadata from remote GitHub. (Terry Fleury) 1.0.4-16 | 2017-09-20 12:39:21 -0500 * Version numbers should be sorted naturally, e.g., 1.10.0 is newer than 1.9.0. (Justin Azoff, Terry Fleury) 1.0.4-14 | 2017-09-20 12:31:59 -0500 * Add option for 'list' command '--nodesc' to suppress description output. (Terry Fleury) 1.0.4-12 | 2017-09-16 13:36:53 -0500 * GH-13: emit an error when a package uses a malformed 'depends' field (Jon Siwek) 1.0.4-11 | 2017-09-16 12:44:50 -0500 * Fix a typo in an error message. (Jon Siwek) 1.0.4-10 | 2017-09-15 13:04:46 -0500 * GH-12: use active branch as package version for local git repos (Jon Siwek) 1.0.4-9 | 2017-09-15 11:47:09 -0500 * GH-10: add a cross-reference for metadata value interpolation in docs (Jon Siwek) 1.0.4-8 | 2017-09-15 11:39:24 -0500 * GH-10: improve docs regarding metadata value interpolation (Jon Siwek) 1.0.4-6 | 2017-09-14 19:52:21 -0500 * Minor quickstart doc simplification. (Jon Siwek) 1.0.4-5 | 2017-09-14 18:56:16 -0500 * Require extra configparser package only for Python < 3.5 where it is not included (Hilko Bengen) * Fix imports for python3 (Robert Haist) 1.0.4-2 | 2017-09-14 15:05:36 -0500 * Add a docutils config file to prevent sphinx/smartypants "smart quotes" Without it, later versions of sphinx are turning quote characters into smart/curly quotes which breaks the generated man page. (Jon Siwek) 1.0.4-1 | 2017-09-14 15:03:04 -0500 * Quickstart doc improvements, add missing links to dependencies (Jon Siwek) 1.0.4 | 2017-07-18 11:41:56 -0500 * Release 1.0.4. 1.0.3-1 | 2017-07-18 11:39:38 -0500 * Give aggregate metadata file a consistent ordering Addresses https://github.com/bro/package-manager/issues/6 (Jon Siwek) 1.0.3 | 2017-06-23 10:17:26 -0500 * Release 1.0.3. 1.0.2-3 | 2017-06-23 10:15:41 -0500 * Install command now doesn't warn/prompt when package has no test_command. (Jon Siwek) * Fix a unit test. (Jon Siwek) * Improve usage documentation: add offline usage example. (Jon Siwek) 1.0.2 | 2017-06-07 22:11:20 -0500 * Release 1.0.2. 1.0.1-1 | 2017-06-07 22:04:55 -0500 * Improve confirmation prompt behavior. Just pressing enter defaults to "yes" and anything besides y/ye/yes (case insensitive) is treated as "no" and aborts. Addresses https://github.com/bro/package-manager/issues/5 (Jon Siwek) 1.0.1 | 2017-06-05 09:54:59 -0500 * Release 1.0.1. 1.0-3 | 2017-06-05 09:54:31 -0500 * Improved handling of packages without scripts. The 'load' and 'unload' commands now return more precise messages if used for packages without scripts. (Jan Grashoefer) 1.0-1 | 2017-02-06 21:02:31 -0600 * Return a proper "bro-config not found" message when running tests. (Jon Siwek) 1.0 | 2017-01-24 19:53:49 -0600 * Release 1.0. 0.9-24 | 2017-01-24 19:53:27 -0600 * Simply directions for uploading to PyPi. (Jon Siwek) 0.9-23 | 2017-01-24 19:41:22 -0600 * Add dependency on btest. (Jon Siwek) 0.9-22 | 2017-01-24 18:09:28 -0600 * Change manager.validate_dependencies API (Jon Siwek) 0.9-21 | 2017-01-21 12:40:09 -0600 * Doc fixes. (Jon Siwek) 0.9-20 | 2017-01-21 12:29:48 -0600 * Run package tests before install/upgrade. (Jon Siwek) * Add 'test' command and 'test_command' metadata field. (Jon Siwek) 0.9-16 | 2017-01-13 18:50:52 -0600 * Fix non-deterministic order of {installed,loaded}_packages() (Jon Siwek) * Add unit tests. (Jon Siwek) * Fix bundle/unbundle --help output. (Jon Siwek) * Don't copy a package's .git or bro-pkg.meta when installing them. (Jon Siwek) 0.9-8 | 2017-01-12 10:23:26 -0600 * Add 'depends' bro-pkg.meta field. (Jon Siwek) * Teach install/upgrade/bundle commands to operate on package dependencies. These commands now construct a dependency graph based on the packages the user has requested plus any already-installed packages and then validates the graph to make sure there are no dependency conflicts. If there are no conflicts, then the set of requested packages get installed/bundled along with any additional packages that were required to satisfy dependencies. For those additional packages, bro-pkg always tries to pick the latest release version that satisfies all dependency specifications. (Jon Siwek) 0.9-2 | 2017-01-09 11:16:43 -0600 * Improve 'info' output for packages w/ empty metadata files. (Jon Siwek) 0.9-1 | 2016-12-09 11:52:23 -0600 * Fix doc typo. (Jon Siwek) 0.9 | 2016-12-08 21:19:02 -0600 * Release 0.9. 0.8-6 | 2016-12-08 21:15:06 -0600 * Fix tracking package versions/changes via branches. A previous change to use shallow git clones broke the tracking of package versioning via git branches. (Jon Siwek) 0.8-5 | 2016-12-08 20:04:53 -0600 * Fix bundle/unbundle automatic lower-casing of urls in bundle manifest (Jon Siwek) 0.8-4 | 2016-12-08 19:50:49 -0600 * Add new bro-pkg.meta field: config_files This allows package authors to specify a list of files that users may directly modify. The upgrade, remove, install, and bundle operations all work in a way that attempts to either preserve a user's local changes or save a backup of their version. (Jon Siwek) 0.8-3 | 2016-12-07 22:54:49 -0600 * BIT-1768: improve handling of packages that have no Bro scripts (Jon Siwek) 0.8-2 | 2016-12-07 20:46:44 -0600 * BIT-1766: add tags() and short_description() to PackageInfo API (Jon Siwek) 0.8-1 | 2016-12-07 20:09:58 -0600 * BIT-1767: improve handling of non-ascii character encoding (Jon Siwek) 0.8 | 2016-11-29 11:21:29 -0600 * Release 0.8. 0.7-19 | 2016-11-29 11:19:17 -0600 * Improve output of info command and allow offline usage. (Jon Siwek) * Update documentation related to metadata aggregation. (Jon Siwek) * Metadata aggregation changes: - bro-pkg.index is now expected to be a simple list of package urls - add --aggregate flag to refresh command that will crawl packages and aggregate their metadata into an aggregate.meta file within local package source clones - add --sources flag to refresh command to explicitly specify operation on a specific set of package sources - add --push flag to refresh command to push local package source aggregate.meta changes to the remote repo (Jon Siwek) 0.7-10 | 2016-11-21 18:04:08 -0600 * Package versioning doc improvements. (Jon Siwek) 0.7-9 | 2016-11-21 12:15:14 -0600 * Update docs regarding package versioning. (Jon Siwek) 0.7-8 | 2016-10-21 10:13:16 -0500 * Using shutil.move instead of os.rename, which works across file systems. (Robin Sommer) 0.7-6 | 2016-10-11 18:54:50 -0500 * BIT-1725: allow bundling on offline machines When requesting to bundle the set of installed packages, they are directly copied into the bundle instead of trying to clone from the remote git repo. (Jon Siwek) 0.7-5 | 2016-10-04 14:05:50 -0500 * BIT-1713: in python3, show usage info when bro-pkg is given no args (Jon siwek) 0.7-4 | 2016-10-03 18:20:57 -0500 * BIT-1716: do shallow git clones of packages to use less space (Jon Siwek) 0.7-3 | 2016-10-03 17:08:31 -0500 * BIT-1715: improve message when failing to clone a package source And make that situation a warning instead of a fatal error. (Jon Siwek) 0.7-2 | 2016-10-03 16:49:24 -0500 * Allow empty values for paths in config file. (Jon Siwek) 0.7-1 | 2016-10-03 15:55:00 -0500 * BIT-1714: expand '~' and env. vars in config file values. And reject relative paths in [paths] section. (Jon Siwek) 0.7 | 2016-09-29 15:26:14 -0500 * Release 0.7. 0.6-14 | 2016-09-29 15:25:44 -0500 * Improve output of 'info' command. (Jon Siwek) 0.6-13 | 2016-09-29 14:59:31 -0500 * Tweak output of bundle/unbundle. (Jon Siwek) 0.6-12 | 2016-09-29 14:43:11 -0500 * Python 3 compatibility fixes. (Jon Siwek) 0.6-11 | 2016-09-28 18:11:44 -0500 * Add 'bundle' and 'unbundle' commands. (Jon Siwek) 0.6-10 | 2016-09-27 13:56:08 -0500 * Fix potential race condition in package installation. (Jon Siwek) 0.6-9 | 2016-09-26 15:31:50 -0500 * Improve formatting of package short descriptions. (Jon Siwek) 0.6-8 | 2016-09-26 15:15:18 -0500 * BIT-1699: improve package validation and format of output. (Jon Siwek) 0.6-7 | 2016-09-26 13:40:26 -0500 * Add a 'purge' command that removes all installed packages. (Jon Siwek) 0.6-6 | 2016-09-21 12:59:01 -0500 * BIT-1699: Add --force flag for install/remove/upgrade. (Jon Siwek) 0.6-5 | 2016-09-21 12:22:18 -0500 * BIT-1699: Add confirmation prompts for install/upgrade/remove. Removed the --all flag for upgrade command. (Jon Siwek) 0.6-4 | 2016-09-21 10:47:53 -0500 * BIT-1697: Improve output format of list/search. (Jon Siwek) 0.6-3 | 2016-09-19 13:29:27 -0500 * BIT-1698: autoconfig command now directly writes to config file. (Jon Siwek) 0.6-2 | 2016-09-19 12:09:52 -0500 * BIT-1697: Add description field to list/search output. (Jon Siwek) 0.6-1 | 2016-09-19 10:52:51 -0500 * BIT-1701: Improve error message when 'install' can't create symlink. (Jon Siwek) 0.6 | 2016-09-14 15:16:30 -0500 * Release 0.6. 0.5-2 | 2016-09-14 15:15:50 -0500 * Improve docs for script_dir/plugin_dir config file options. (Jon Siwek) 0.5-1 | 2016-09-14 15:09:12 -0500 * Improve automatic relocation of script_dir. (Jon Siwek) 0.5 | 2016-09-05 15:03:14 -0500 * Release 0.5. 0.4-2 | 2016-09-05 15:01:38 -0500 * Improve error message for insufficient permissions (Jon Siwek) 0.4-1 | 2016-09-05 14:09:55 -0500 * Improve output of 'info' command. (Jon Siwek) 0.4 | 2016-08-16 16:55:48 -0500 * Release 0.4. 0.3-4 | 2016-08-16 16:55:20 -0500 * Docs: add 'autoconfig' command (Jon Siwek) 0.3-3 | 2016-08-16 16:10:21 -0500 * Add 'autoconfig' command, remove 'bro_exe' option. bro-pkg can now use bro-config to automatically determine paths. (Jon Siwek) 0.3-2 | 2016-08-15 13:32:18 -0500 * Docs: clarifications (Jon Siwek) 0.3-1 | 2016-08-12 16:20:30 -0500 * Docs: add bro.org link in README (Jon Siwek) 0.3 | 2016-08-12 15:54:18 -0500 * Release 0.3. 0.2-47 | 2016-08-12 15:53:13 -0500 * Docs: add versioning/release guide for developers (Jon Siwek) 0.2-46 | 2016-08-11 19:45:21 -0500 * setup.py packaging improvements (Jon Siwek) 0.2-45 | 2016-08-11 14:35:06 -0500 * Docs: fix sphinx-argparse 'nosubcommands' option (Jon Siwek) 0.2-44 | 2016-08-11 14:16:22 -0500 * Docs: adjust admonitions in README (Jon Siwek) 0.2-43 | 2016-08-11 14:09:44 -0500 * Docs: organize external links (Jon Siwek) 0.2-42 | 2016-08-10 19:58:15 -0500 * Fix `list all` to show installed packages that don't come from a source (Jon Siwek) 0.2-41 | 2016-08-10 19:35:53 -0500 * Docs: add package creation walkthroughs (Jon Siwek) * Don't load packages unless they have a __load__.bro (Jon Siwek) * Improve handling of relative paths in package urls (Jon Siwek) 0.2-38 | 2016-08-10 14:29:30 -0500 * Teach search command to inspect package index tags (Jon Siwek) 0.2-37 | 2016-08-09 18:55:30 -0500 * Delay printing installation activity ticks. (Jon Siwek) 0.2-36 | 2016-08-09 18:41:54 -0500 * Change package sources to use index files instead of git submodules. (Jon Siwek) 0.2-35 | 2016-08-09 13:54:46 -0500 * Install packages in a thread. So it doesn't look like bro-pkg is hanging when installing a package that has to be built. (Jon Siwek) 0.2-34 | 2016-08-09 13:06:24 -0500 * Using ``install --version`` now also pins the package. (Jon Siwek) 0.2-33 | 2016-08-08 16:19:59 -0500 * info command can now take a --version flag. (Jon Siwek) 0.2-32 | 2016-08-08 15:16:54 -0500 * Fix install command to use the right version of bro-pkg.meta. (Jon Siwek) 0.2-31 | 2016-08-08 12:47:02 -0500 * Change config/metadata file option names. (Jon Siwek) bro-pkg.meta: scriptpath -> script_dir pluginpath -> plugin_dir buildcmd -> build_command bro-pkg config file: statedir -> state_dir scriptdir -> script_dir plugindir -> plugin_dir 0.2-30 | 2016-08-07 13:45:00 -0500 * Rename pkg.meta to bro-pkg.meta. (Jon Siwek) 0.2-29 | 2016-08-07 13:08:00 -0500 * upgrade command: require --all when omitting package list. (Jon Siwek) 0.2-28 | 2016-08-07 12:51:20 -0500 * Docs: adjust bro-pkg --help output. (Jon Siwek) 0.2-27 | 2016-08-07 12:35:40 -0500 * Add warnings to not directly modify certain files/dirs. (Jon Siwek) 0.2-26 | 2016-08-07 12:05:40 -0500 * Docs: show actual version in sidebar. (Jon Siwek) 0.2-25 | 2016-08-06 15:08:12 -0500 * Docs: don't restrict argparse-generated table widths. (Jon Siwek) * Docs: misc. style fixes and clarifications. (Jon Siwek) * Docs: switch theme (again) to Read the Docs theme. 0.2-22 | 2016-08-06 13:12:26 -0500 * Docs: clarifications, organization, style. (Jon Siwek) 0.2-21 | 2016-08-05 13:41:47 -0500 * Docs: change theme. (Jon Siwek) * Docs: GitPython and semantic_version build requirements (Jon Siwek) 0.2-19 | 2016-08-04 13:25:48 -0500 * Docs: add requirements file (Jon Siwek) 0.2-18 | 2016-08-03 16:10:18 -0500 * Docs: fix Sphinx argparse extension to check for outdated modules. Sphinx itself should now always be able to determine whether the command-line bro-pkg documentation needs to be rebuilt. (Jon Siwek) 0.2-17 | 2016-08-02 14:46:16 -0500 * Docs: rebuild docs on version string change. (Jon Siwek) 0.2-16 | 2016-08-02 14:34:49 -0500 * Docs: give each bro-pkg command its own section (Jon Siwek) 0.2-15 | 2016-08-01 14:34:06 -0500 * Documentation organization/layout/theming. (Jon Siwek) 0.2-14 | 2016-07-24 14:55:28 -0500 * Clarification to installation doc. (Jon Siwek) 0.2-13 | 2016-07-24 12:18:40 -0500 * Add a 'gh-pages' Makefile target. (Jon Siwek) * Add .nojekyll file to built sphinx docs. (Jon Siwek) 0.2-11 | 2016-07-23 15:32:24 -0500 * Add installation/setup docs. (Jon Siwek) 0.2-10 | 2016-07-23 14:19:09 -0500 * Add a prebuilt man page. (Jon Siwek) 0.2-9 | 2016-07-22 15:07:57 -0500 * Change bro-pkg to not require a config file and add setup.py. (Jon Siwek) 0.2-8 | 2016-07-21 12:14:53 -0500 * Improve speed of checking for changed remote package sources. (Jon Siwek) * Improve how 'info' command locates packages. (Jon Siwek) * Emit an error when package has non-existant 'scriptpath'. (Jon Siwek) * Documentation clarifications. (Jon Siwek) 0.2-4 | 2016-07-21 09:49:15 -0500 * Make output of 'info' command have a stable order. (Jon Siwek) 0.2-3 | 2016-07-21 09:44:42 -0500 * Python3 compatibility fixes. (Jon Siwek) 0.2-2 | 2016-07-20 14:35:38 -0500 * Documentation and other clean up. Sphinx is used docs: manual, api reference, man page. (Jon Siwek) 0.2-1 | 2016-07-14 12:26:56 -0500 * Fix a mistake in --help output. (Jon Siwek) 0.2 | 2016-07-14 10:51:11 -0500 * Release 0.2. package-manager-3.0.1/README0000777000175000017500000000000014565163601016655 2doc/overview.rstustar rharhapackage-manager-3.0.1/.git-blame-ignore-revs0000644000175000017500000000010714565163601016731 0ustar rharha# Migrate code style to Black 99520f9201340d57d2cc9cf6f45992b994eff7c4 package-manager-3.0.1/setup.cfg0000644000175000017500000000070314565163601014454 0ustar rharha[bdist_wheel] universal = 1 [flake8] max_line_length = 100 # E203: whitespace before ':' (black / flake8 disagreement) # W503: line break before binary operator (black / flake8 disagreement) ignore=E203,W503 # E402: module level import not at top of file # F405: may be undefined, or defined from star imports # F403: from .manager import *' used; unable to detect undefined names per-file-ignores = zeekpkg/__init__.py: F405,F403,E402 zkg: E402