././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1759517099.1824224 cvdupdate-1.2.0/0000755000175100017510000000000015070014653013140 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/LICENSE0000644000175100017510000002613515070014643014153 0ustar00runnerrunner Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1759517099.1824224 cvdupdate-1.2.0/PKG-INFO0000644000175100017510000003726015070014653014245 0ustar00runnerrunnerMetadata-Version: 2.4 Name: cvdupdate Version: 1.2.0 Summary: ClamAV Private Database Mirror Updater Tool Home-page: https://github.com/Cisco-Talos/cvdupdate Author: The ClamAV Team Author-email: clamav-bugs@external.cisco.com Classifier: Programming Language :: Python :: 3 Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: OS Independent Description-Content-Type: text/markdown License-File: LICENSE Requires-Dist: click>=7.0 Requires-Dist: colorlog>=6.7 Requires-Dist: colorama Requires-Dist: requests Requires-Dist: dnspython>=2.1.0 Requires-Dist: rangehttpserver Requires-Dist: packaging Dynamic: author Dynamic: author-email Dynamic: classifier Dynamic: description Dynamic: description-content-type Dynamic: home-page Dynamic: license-file Dynamic: requires-dist Dynamic: summary

A tool to download and update clamav databases and database patch files for the purposes of hosting your own database mirror.

Copyright (C) 2021-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.

PyPI version PyPI - Python Version

## About This tool downloads the latest ClamAV databases along with the latest database patch files. This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again. ## Requirements - Python 3.6 or newer. - An internet connection with DNS enabled. - The following Python packages. These will be installed automatically if you use `pip`, but may need to be installed manually otherwise: - `click` v7.0 or newer - `colorlog` v6.7 or newer - `colorama` - `requests` - `dnspython` v2.1.0 or newer - `rangehttpserver` ## Installation You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally. Install `cvdupdate` from PyPI: ```bash python3 -m pip install --user cvdupdate ``` ### Updating Your Installation When running `cvd update` to update the databases, it will also check if there is a new version of the `cvdupdate` package on Python's PyPI package repository. If there is a newer version of `cvdupdate`, you will see a message prompting you to upgrade. It will look someething like this: ``` WARNING You are running cvdupdate version: 1.1.0. WARNING There is a newer version on PyPI: 1.1.1. Please update! ``` To upgrade the `cvdupdate` package through PyPI, run: ```bash python3 -m pip install --user --upgrade cvdupdate ``` ## Basic Usage Use the `--help` option with any `cvd` command to get help. ```bash cvd --help ``` > _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this: > > ```bash > python -m cvdupdate --help > ``` (optional) You may wish to customize where the databases are stored: ```bash cvd config set --dbdir ``` Run this to download the latest database and associated CDIFF patch files: ```bash cvd update ``` Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory. Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation. Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam. ```bash cvd serve ``` > _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work. Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace: ``` DatabaseMirror database.clamav.net ``` ... with: ``` DatabaseMirror http://localhost:8000 ``` > _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template. Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server. Run `cvd update` as often as you need. Maybe put it in a `cron` job. > _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood. ### Cron Example Cron is a popular choice to automate frequent tasks on Linux / Unix systems. 1. Open a terminal running as the user which you want CVD-Update to run under, do the following: ```bash crontab -e ``` 2. Press `i` to insert new text, and add this line: ```bash 30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null" ``` Or instead of `~/`, you can do this, replacing `username` with your user name: ```bash 30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null" ``` 3. Press , then type `:wq` and press to write the file to disk and quit. **About these settings**: I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue. CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs. ## Optional Functionality ### Using a custom DNS server DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available 1. Set the nameserver in the config. Eg: ```bash cvd config set --nameserver 208.67.222.222 ``` 2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg: ```bash CVDUPDATE_NAMESERVER="208.67.222.222" cvd update ``` The environment variable will take precedence over the nameserver config setting. Note: Both options can be used to provide a comma-delimited list of nameservers to utilize for resolution. ### Using a proxy Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this: First, set a custom domain name server to use the proxy: ```bash cvd config set --nameserver ``` Then run CVD-Update like this: ```bash http_proxy=http://: https_proxy=http://: cvd update -V ``` Or create a script to wrap the CVD-Update call. Something like: ```bash #!/bin/bash http_proxy=http://: export http_proxy https_proxy=http://: export https_proxy cvd update -V ``` > _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details. > > Adding support for proxy authentication is a ripe opportunity for a community contribution to the project. ## Files and directories created by CVD-Update This tool is to creates the following directories: - `~/.cvdupdate` - `~/.cvdupdate/logs` - `~/.cvdupdate/databases` This tool creates the following files: - `~/.cvdupdate/config.json` - `~/.cvdupdate/state.json` - `~/.cvdupdate/databases/.cvd` - `~/.cvdupdate/databases/-.cdiff` - `~/.cvdupdate/logs/.log` > _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory. > _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it. ## Additional Usage ### Get familiar with the tool Familiarize yourself with the various commands using the `--help` option. ```bash cvd --help cvd config --help cvd update --help cvd add --help cvd clean --help ``` Print out the current list of databases. ```bash cvd list -V ``` Print out the config to see what it looks like. ```bash cvd config show ``` ### Do an update Do an update, use "verbose mode" to so you can get a feel for how it works. ```bash cvd update -V ``` List out the databases again: ```bash cvd list -V ``` The print out the config again so you can see what's changed. ```bash cvd config show ``` And maybe take a peek in the database directory as well to see it for yourself. ```bash ls ~/.cvdupdate/database ``` Have a look at the logs if you wish. ```bash ls ~/.cvdupdate/logs cat ~/.cvdupdate/logs/* ``` ### Add an additional database Maybe add an additional database that is not part of the default set of databases. ```bash cvd add linux.cvd https://database.clamav.net/linux.cvd ``` List out the databases again: ```bash cvd list -V ``` ### Serve it up, Test out FreshClam Test out your mirror with FreshClam on the same computer. This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000). You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with: ``` DatabaseMirror http://localhost:8000 ``` ## Use docker Build docker image ```bash docker build . --tag cvdupdate:latest ``` Run image, that will automaticly update databases in folder `/srv/cvdupdate` and write logs to `/var/log/cvdupdate` ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ cvdupdate:latest ``` Run image, that will automaticly update databases in folder `/srv/cvdupdate`, write logs to `/var/log/cvdupdate` and set owner of files to user with ID 1000 ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ -e USER_ID=1000 \ cvdupdate:latest ``` Default update interval is `30 */4 * * *` (see [Cron Example](#cron-example)) You may pass custom update interval in environment variable `CRON` For example - update every day in 00:00 ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ -e CRON='0 0 * * *' \ cvdupdate:latest ``` ## Use Docker Compose A Docker `compose.yaml` is provided to: 1. Regularly update a Docker volume with the latest ClamAV databases. 2. Serve a database mirror on port 8000 using the Apache webserver. Edit the `compose.yaml` file if you need to change the default values: * Port 8000 * USER_ID=0 * CRON=30 */4 * * * ### Build ```bash docker compose build ``` ### Start ```bash docker compose up -d ``` ### Stop ```bash docker compose down ``` ### Volumes Volumes are defined in `compose.yaml` and will be auto-created when you run `docker compose up` ``` DRIVER VOLUME NAME local cvdupdate_database local cvdupdate_log ``` ## Contribute We'd love your help. There are many ways to contribute! ### Community Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q). ### Report issues If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue. ### Development If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated. If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature. _By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._ #### Development Set-up The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started. 1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory. For example: ```bash git clone https://github.com//cvdupdate.git ``` 2. Make sure CVD-Update is not already installed. If it is, remove it. ```bash python3 -m pip uninstall cvdupdate ``` 3. Use pip to install CVD-Update in "edit" mode. ```bash python3 -m pip install -e --user ./cvdupdate ``` Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands. ### Conduct This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated. ## License CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License. A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0). Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/README.md0000644000175100017510000003555715070014643014435 0ustar00runnerrunner

A tool to download and update clamav databases and database patch files for the purposes of hosting your own database mirror.

Copyright (C) 2021-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.

PyPI version PyPI - Python Version

## About This tool downloads the latest ClamAV databases along with the latest database patch files. This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again. ## Requirements - Python 3.6 or newer. - An internet connection with DNS enabled. - The following Python packages. These will be installed automatically if you use `pip`, but may need to be installed manually otherwise: - `click` v7.0 or newer - `colorlog` v6.7 or newer - `colorama` - `requests` - `dnspython` v2.1.0 or newer - `rangehttpserver` ## Installation You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally. Install `cvdupdate` from PyPI: ```bash python3 -m pip install --user cvdupdate ``` ### Updating Your Installation When running `cvd update` to update the databases, it will also check if there is a new version of the `cvdupdate` package on Python's PyPI package repository. If there is a newer version of `cvdupdate`, you will see a message prompting you to upgrade. It will look someething like this: ``` WARNING You are running cvdupdate version: 1.1.0. WARNING There is a newer version on PyPI: 1.1.1. Please update! ``` To upgrade the `cvdupdate` package through PyPI, run: ```bash python3 -m pip install --user --upgrade cvdupdate ``` ## Basic Usage Use the `--help` option with any `cvd` command to get help. ```bash cvd --help ``` > _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this: > > ```bash > python -m cvdupdate --help > ``` (optional) You may wish to customize where the databases are stored: ```bash cvd config set --dbdir ``` Run this to download the latest database and associated CDIFF patch files: ```bash cvd update ``` Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory. Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation. Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam. ```bash cvd serve ``` > _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work. Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace: ``` DatabaseMirror database.clamav.net ``` ... with: ``` DatabaseMirror http://localhost:8000 ``` > _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template. Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server. Run `cvd update` as often as you need. Maybe put it in a `cron` job. > _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood. ### Cron Example Cron is a popular choice to automate frequent tasks on Linux / Unix systems. 1. Open a terminal running as the user which you want CVD-Update to run under, do the following: ```bash crontab -e ``` 2. Press `i` to insert new text, and add this line: ```bash 30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null" ``` Or instead of `~/`, you can do this, replacing `username` with your user name: ```bash 30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null" ``` 3. Press , then type `:wq` and press to write the file to disk and quit. **About these settings**: I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue. CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs. ## Optional Functionality ### Using a custom DNS server DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available 1. Set the nameserver in the config. Eg: ```bash cvd config set --nameserver 208.67.222.222 ``` 2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg: ```bash CVDUPDATE_NAMESERVER="208.67.222.222" cvd update ``` The environment variable will take precedence over the nameserver config setting. Note: Both options can be used to provide a comma-delimited list of nameservers to utilize for resolution. ### Using a proxy Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this: First, set a custom domain name server to use the proxy: ```bash cvd config set --nameserver ``` Then run CVD-Update like this: ```bash http_proxy=http://: https_proxy=http://: cvd update -V ``` Or create a script to wrap the CVD-Update call. Something like: ```bash #!/bin/bash http_proxy=http://: export http_proxy https_proxy=http://: export https_proxy cvd update -V ``` > _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details. > > Adding support for proxy authentication is a ripe opportunity for a community contribution to the project. ## Files and directories created by CVD-Update This tool is to creates the following directories: - `~/.cvdupdate` - `~/.cvdupdate/logs` - `~/.cvdupdate/databases` This tool creates the following files: - `~/.cvdupdate/config.json` - `~/.cvdupdate/state.json` - `~/.cvdupdate/databases/.cvd` - `~/.cvdupdate/databases/-.cdiff` - `~/.cvdupdate/logs/.log` > _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory. > _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it. ## Additional Usage ### Get familiar with the tool Familiarize yourself with the various commands using the `--help` option. ```bash cvd --help cvd config --help cvd update --help cvd add --help cvd clean --help ``` Print out the current list of databases. ```bash cvd list -V ``` Print out the config to see what it looks like. ```bash cvd config show ``` ### Do an update Do an update, use "verbose mode" to so you can get a feel for how it works. ```bash cvd update -V ``` List out the databases again: ```bash cvd list -V ``` The print out the config again so you can see what's changed. ```bash cvd config show ``` And maybe take a peek in the database directory as well to see it for yourself. ```bash ls ~/.cvdupdate/database ``` Have a look at the logs if you wish. ```bash ls ~/.cvdupdate/logs cat ~/.cvdupdate/logs/* ``` ### Add an additional database Maybe add an additional database that is not part of the default set of databases. ```bash cvd add linux.cvd https://database.clamav.net/linux.cvd ``` List out the databases again: ```bash cvd list -V ``` ### Serve it up, Test out FreshClam Test out your mirror with FreshClam on the same computer. This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000). You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with: ``` DatabaseMirror http://localhost:8000 ``` ## Use docker Build docker image ```bash docker build . --tag cvdupdate:latest ``` Run image, that will automaticly update databases in folder `/srv/cvdupdate` and write logs to `/var/log/cvdupdate` ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ cvdupdate:latest ``` Run image, that will automaticly update databases in folder `/srv/cvdupdate`, write logs to `/var/log/cvdupdate` and set owner of files to user with ID 1000 ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ -e USER_ID=1000 \ cvdupdate:latest ``` Default update interval is `30 */4 * * *` (see [Cron Example](#cron-example)) You may pass custom update interval in environment variable `CRON` For example - update every day in 00:00 ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ -e CRON='0 0 * * *' \ cvdupdate:latest ``` ## Use Docker Compose A Docker `compose.yaml` is provided to: 1. Regularly update a Docker volume with the latest ClamAV databases. 2. Serve a database mirror on port 8000 using the Apache webserver. Edit the `compose.yaml` file if you need to change the default values: * Port 8000 * USER_ID=0 * CRON=30 */4 * * * ### Build ```bash docker compose build ``` ### Start ```bash docker compose up -d ``` ### Stop ```bash docker compose down ``` ### Volumes Volumes are defined in `compose.yaml` and will be auto-created when you run `docker compose up` ``` DRIVER VOLUME NAME local cvdupdate_database local cvdupdate_log ``` ## Contribute We'd love your help. There are many ways to contribute! ### Community Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q). ### Report issues If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue. ### Development If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated. If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature. _By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._ #### Development Set-up The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started. 1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory. For example: ```bash git clone https://github.com//cvdupdate.git ``` 2. Make sure CVD-Update is not already installed. If it is, remove it. ```bash python3 -m pip uninstall cvdupdate ``` 3. Use pip to install CVD-Update in "edit" mode. ```bash python3 -m pip install -e --user ./cvdupdate ``` Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands. ### Conduct This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated. ## License CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License. A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0). Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1759517099.1804225 cvdupdate-1.2.0/cvdupdate/0000755000175100017510000000000015070014653015117 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/cvdupdate/__init__.py0000644000175100017510000000000015070014643017215 0ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/cvdupdate/__main__.py0000644000175100017510000002347715070014643017225 0ustar00runnerrunner#!/usr/bin/env python3 """ CVD-Update: ClamAV Database Updater """ _description = """ A tool to download and update clamav databases and database patch files for the purposes of hosting your own database mirror. """ _copyright = """ Copyright (C) 2021-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved. """ """ Author: Micah Snyder Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. """ import logging import os import sys from pathlib import Path import click import colorlog try: from importlib.metadata import PackageNotFoundError, version as _get_version except ImportError: # pragma: no cover - backport for older Pythons from importlib_metadata import PackageNotFoundError, version as _get_version from http.server import HTTPServer from RangeHTTPServer import RangeRequestHandler from cvdupdate import auto_updater from cvdupdate.cvdupdate import CVDUpdate handler = colorlog.StreamHandler() handler.setFormatter( colorlog.ColoredFormatter( "%(log_color)s%(asctime)s %(name)s %(levelname)s %(message)s" ) ) logging.basicConfig(level=logging.DEBUG, handlers=[handler]) module_logger = logging.getLogger("cvdupdate") module_logger.setLevel(logging.DEBUG) from colorama import Fore, Back, Style def _package_version() -> str: try: return _get_version('cvdupdate') except PackageNotFoundError: return '0.0' # # CLI Interface # @click.group( epilog=Fore.BLUE + __doc__ + "\n" + Fore.GREEN + _description + "\n" + f"\nVersion {_package_version()}\n" + Style.RESET_ALL + _copyright, ) def cli(): pass @cli.command("list") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") def db_list(config: str, verbose: bool): """ List the DBs found in the database directory. """ m = CVDUpdate(config=config, verbose=verbose) m.db_list() @cli.command("show") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.argument("db", required=True) def db_show(config: str, verbose: bool, db: str): """ Show details about a specific database. """ m = CVDUpdate(config=config, verbose=verbose) if not m.db_show(db): sys.exit(1) @cli.command("update") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.option("--debug-mode", "-D", is_flag=True, default=False, help="Print out HTTP headers for debugging purposes. [optional]") @click.argument("db", required=False, default="") def db_update(config: str, verbose: bool, db: str, debug_mode: bool): """ Update the DBs from the internet. Will update all DBs if DB not specified. """ m = CVDUpdate(config=config, verbose=verbose) errors = m.db_update(db, debug_mode) if errors > 0: sys.exit(errors) @cli.command("add") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.argument("db", required=True) @click.argument("url", required=True) def db_add(config: str, verbose: bool, db: str, url: str): """ Add a db to the list of known DBs. """ m = CVDUpdate(config=config, verbose=verbose) if not m.config_add_db(db, url=url): sys.exit(1) @cli.command("remove") @click.option("--config", "-c", type=str, required=False, default="") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.argument("db", required=True) def db_remove(config: str, verbose: bool, db: str): """ Remove a db from the list of known DBs and delete local copies of the DB. """ m = CVDUpdate(config=config, verbose=verbose) if not m.config_remove_db(db): sys.exit(1) @cli.group(help="Commands to configure.") def config(): pass @config.command("set") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.option("--logdir", "-l", type=click.Path(), required=False, default="", help="Set a custom log directory. [optional]") @click.option("--dbdir", "-d", type=click.Path(), required=False, default="", help="Set a custom database directory. [optional]") @click.option("--nameserver", "-n", type=click.STRING, required=False, default="", help="Set a custom DNS nameserver. [optional]") def config_set(config: str, verbose: bool, logdir: str, dbdir: str, nameserver: str): """ Set up first time configuration. The default directories will be in ~/.cvdupdate """ CVDUpdate( config=config, verbose=verbose, log_dir=logdir, db_dir=dbdir, nameserver=nameserver) @config.command("show") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") def config_show(config: str, verbose: bool): """ Print out the current configuration. """ m = CVDUpdate(config=config, verbose=verbose) m.config_show() @cli.group(help="Commands to clean up.") def clean(): pass @clean.command("dbs") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") def clean_dbs(config: str, verbose: bool): """ Delete all files in the database directory. """ m = CVDUpdate(config=config, verbose=verbose) m.clean_dbs() @clean.command("logs") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") def clean_logs(config: str, verbose: bool): """ Delete all files in the logs directory """ m = CVDUpdate(config=config, verbose=verbose) m.clean_logs() @clean.command("all") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") def clean_all(config: str, verbose: bool): """ Delete the logs, databases, and config file. """ m = CVDUpdate(config=config, verbose=verbose) m.clean_all() @cli.command("serve") @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.option("--update-interval-seconds", "-U", type=click.INT, required=False, default=0, help="Time in seconds before the next database update") @click.argument("port", type=int, required=False, default=8000) def serve(port: int, config: str, verbose: bool, update_interval_seconds: int): """ Serve up the database directory. Not a production quality server. Intended for testing purposes. """ m = CVDUpdate(config=config, verbose=verbose) os.chdir(str(m.db_dir)) m.logger.info(f"Serving up {m.db_dir} on localhost:{port}...") auto_updater.start(update_interval_seconds) RangeRequestHandler.protocol_version = 'HTTP/1.0' # TODO(danvk): pick a random, available port httpd = HTTPServer(('', port), RangeRequestHandler) httpd.serve_forever() # # Command Aliases # @cli.command("list") @click.pass_context @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") def list_alias(ctx, config: str, verbose: bool): """ List the DBs found in the database directory. This is just an alias for `db list`. """ ctx.forward(db_list) @cli.command("show") @click.pass_context @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.argument("db", required=True) def show_alias(ctx, config: str, verbose: bool, db: str): """ Show details about a specific database. This is just an alias for `db show`. """ ctx.forward(db_show) @cli.command("update") @click.pass_context @click.option("--config", "-c", type=click.Path(), required=False, default="", help="Config path. [optional]") @click.option("--verbose", "-V", is_flag=True, default=False, help="Verbose output. [optional]") @click.option("--debug-mode", "-D", is_flag=True, default=False, help="Print out HTTP headers for debugging purposes. [optional]") @click.argument("db", required=False, default="") def update_alias(ctx, config: str, verbose: bool, db: str, debug_mode: bool): """ Update local copy of DBs. This is just an alias for `db show`. """ ctx.forward(db_update) if __name__ == "__main__": sys.argv[0] = "cvdupdate" cli(sys.argv[1:]) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/cvdupdate/auto_updater.py0000644000175100017510000000154515070014643020171 0ustar00runnerrunnerfrom threading import Event, Thread from cvdupdate.cvdupdate import CVDUpdate def start(interval: int) -> None: """Spawn a thread to update the AV db after "interval" seconds :param interval: the interval in seconds between 2 updates of the db """ if interval > 0: Thread(target=_update, daemon=True, args=[interval]).start() def _update(interval: int) -> None: """Don't call this directly Updates the AV db after every "interval" seconds when it was started :param interval: the interval in seconds between 2 updates of the db """ ticker = Event() m = CVDUpdate() m.logger.info(f"Updating the database every {interval} seconds") while not ticker.wait(interval): errors = m.db_update(debug_mode=True) if errors > 0: m.logger.error("Failed to fetch updates from ClamAV databases") ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/cvdupdate/cvdupdate.py0000644000175100017510000015141015070014643017451 0ustar00runnerrunner""" Copyright (C) 2021-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved. This module provides a tool to download and update clamav databases and database patch files (CDIFFs) for the purposes of hosting your own database mirror. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. """ import copy import datetime import http.client as http_client import json import logging import os import platform import re import subprocess import sys import time import uuid from enum import Enum from pathlib import Path from typing import * try: from importlib.metadata import PackageNotFoundError, version as _get_version except ImportError: # pragma: no cover - backport for older Pythons from importlib_metadata import PackageNotFoundError, version as _get_version try: import requests except ModuleNotFoundError: class _RequestsMissing: def __getattr__(self, name): raise ModuleNotFoundError( "The 'requests' package is required to perform network operations. " "Install it with 'pip install requests'." ) requests = _RequestsMissing() try: from dns import resolver except ModuleNotFoundError: class _DNSMissing: def __getattr__(self, name): raise ModuleNotFoundError( "The 'dnspython' package is required for DNS lookups. " "Install it with 'pip install dnspython'." ) resolver = _DNSMissing() from packaging import version class CvdStatus(Enum): NO_UPDATE = 0 UPDATED = 1 ERROR = 2 class CVDUpdate: default_config_path: Path = Path.home() / ".cvdupdate" / "config.json" default_config: dict = { "nameserver" : "", "max retry" : 3, # No `cvd config set` option to set this, because we don't # _really_ want people hammering the CDN with partial downloads. "log directory" : str(Path.home() / ".cvdupdate" / "logs"), "rotate logs" : True, "# logs to keep" : 30, "db directory" : str(Path.home() / ".cvdupdate" / "database"), "rotate cdiffs" : True, "# cdiffs to keep" : 30, "state file": "", } default_state: dict = { "dbs" : { "main.cvd" : { "url" : "https://database.clamav.net/main.cvd", "retry after" : 0, "last modified" : 0, "last checked" : 0, "DNS field" : 1, # only for CVDs "local version" : 0, # only for CVDs "CDIFFs" : [] # only for CVDs }, "daily.cvd" : { "url" : "https://database.clamav.net/daily.cvd", "retry after" : 0, "last modified" : 0, "last checked" : 0, "DNS field" : 2, "local version" : 0, "CDIFFs" : [] }, "bytecode.cvd" : { "url" : "https://database.clamav.net/bytecode.cvd", "retry after" : 0, "last modified" : 0, "last checked" : 0, "DNS field" : 7, "local version" : 0, "CDIFFs" : [] }, }, } config_path: Path config: dict state: dict db_dir: Path log_dir: Path version: str def __init__( self, config: str = "", log_dir: str = "", db_dir: str = "", nameserver: str = "", verbose: bool = False, ) -> None: """ CVDUpdate class. Args: log_dir: path output log. db_dir: path where databases will be downloaded. verbose: Enable DEBUG-level logs and other verbose messages. """ try: self.version = _get_version('cvdupdate') except PackageNotFoundError: self.version = "0.0" self.verbose = verbose self._read_config( config, db_dir, log_dir, nameserver) self._init_logging() def _init_logging(self) -> None: """ Initializes the logging parameters. """ today = datetime.datetime.now() log_file = self.log_dir / f"{today:%Y-%m-%d}.log" if not self.log_dir.exists(): # Make a new log directory os.makedirs(log_file.parent) else: # Log dir already exists, lets check if we need to prune old logs logs = self.log_dir.glob('*.log') for log in logs: log_date_str = str(log.stem) log_date = datetime.datetime.strptime(log_date_str, "%Y-%m-%d") if log_date + datetime.timedelta(days=self.config["# logs to keep"]) < today: # Log is too old, delete! os.remove(str(log)) stderr_level = logging.WARNING stderr_handler = logging.StreamHandler(sys.stderr) stderr_handler.setLevel(stderr_level) class FilterNotStdErr(logging.Filter): def filter(self, record: logging.LogRecord) -> bool: return record.levelno < stderr_level stdout_handler = logging.StreamHandler(sys.stdout) stdout_handler.addFilter(FilterNotStdErr()) logging.basicConfig( level=logging.DEBUG if self.verbose else logging.INFO, format="%(asctime)s - %(levelname)s: %(message)s", datefmt="%Y-%m-%d %I:%M:%S %p", force=True, # an import might already have the root logger configured handlers=[ stderr_handler, stdout_handler, logging.FileHandler(log_file), ], ) self.logger = logging.getLogger(f"cvdupdate-{self.version}") # Also set the log level for urllib3, because it's "DEBUG" by default, # and we may not like that. urllib3_logger = logging.getLogger("urllib3.connectionpool") urllib3_logger.setLevel(self.logger.level) def _read_config(self, config: str, db_dir: str, log_dir: str, nameserver: str) -> None: """ Read in the config file. Create a new one if one does not already exist. """ need_save = False if config == "": self.config_path = copy.deepcopy(self.default_config_path) else: self.config_path = Path(config) if self.config_path.exists(): # Config already exists, load it. with self.config_path.open('r') as config_file: self.config = json.load(config_file) else: # Config does not exist, use default self.config = copy.deepcopy(self.default_config) need_save = True if db_dir != "": self.config["db directory"] = db_dir need_save = True self.db_dir = Path(self.config["db directory"]) if log_dir != "": self.config["log directory"] = log_dir need_save = True self.log_dir = Path(self.config["log directory"]) if nameserver != "": self.config['nameserver'] = nameserver need_save = True # For backwards compatibility with older configs. if 'nameserver' not in self.config: self.config['nameserver'] = "" need_save = True if 'max retry' not in self.config: self.config['max retry'] = 3 need_save = True if not hasattr(self, 'state'): self.state = {} # keep database state in a separate file, defaulting to same dir as config file if 'state file' not in self.config or self.config['state file'] == '': self.config['state file'] = str(self.config_path.parent / "state.json") need_save = True # handle migration from config.json to state.json if 'dbs' in self.config: self.state['dbs'] = self.config['dbs'] del self.config['dbs'] if 'uuid' in self.config: self.state['uuid'] = self.config['uuid'] del self.config['uuid'] state_file = Path(self.config['state file']) if state_file.exists(): # state file exists, load it. with state_file.open('r') as st_fi: self.state = json.load(st_fi) elif self.state == {} or 'dbs' not in self.state: # state file does not exist # so we either have a fresh install or we have a messed up json # create a skeleton structure self.state = copy.deepcopy(self.default_state) need_save = True if 'uuid' not in self.state: # Create a UUID to put in our User-Agent for better (anonymous) metrics self.state['uuid'] = str(uuid.uuid4()) need_save = True if need_save: self._save_config() def _save_config(self) -> None: """ Save the current configuration. """ for fi in (self.config_path, Path(self.config['state file'])): if not fi.parent.exists(): # parent directory doesn't exist yet try: os.makedirs(str(fi.parent)) except Exception as exc: print("Failed to create config directory!") raise exc try: with self.config_path.open('w') as config_file: json.dump(self.config, config_file, indent=4) except Exception as exc: print("Failed to create config file!") raise exc try: with open(self.config['state file'], 'w') as state_file: json.dump(self.state, state_file, indent=4) except Exception as exc: print("Failed to create state file!") raise exc if self.verbose: print(f"Saved: {self.config_path}\n") print(f"Saved: {self.config['state file']}\n") def config_show(self): """ Print out the config """ print(f"Config file: {self.config_path}\n") print(f"Config:\n{json.dumps(self.config, indent=4)}\n") print(f"State file: {self.config['state file']}\n") print(f"State:\n{json.dumps(self.state, indent=4)}\n") def update(self, db: str = "") -> bool: """ Update a specific database or all the databases. """ def clean_dbs(self): """ Delete cvd controlled files in the database directory. """ dbs = self.state['dbs'].keys() for db in dbs: cvddb = self.db_dir / db if cvddb.exists(): try: self.logger.info(f"Deleting: {db}") os.remove(str(cvddb)) # If there is a matching .sign digital signature file, remove it too if os.path.exists(str(cvddb) + ".sign"): os.remove(str(cvddb) + ".sign") except Exception as exc: self.logger.debug(f"Tried to remove {db}") raise exc # Remove / clear all CDIFFs cdiff_files = self.db_dir.glob('*.cdiff') for cdiff in cdiff_files: try: self.logger.info(f"Deleting CDIFF: {cdiff.name}") os.remove(str(cdiff)) # If there is a matching .sign digital signature file, remove it too if os.path.exists(str(cdiff) + ".sign"): os.remove(str(cdiff) + ".sign") except Exception as exc: self.logger.debug(f"Tried to remove CDIFFs.") # Config cleanup for db in dbs: self.state['dbs'][db]['CDIFFs'] = [] self.state['dbs'][db]['last modified'] = 0 self.state['dbs'][db]['last checked'] = 0 self.state['dbs'][db]['local version'] = 0 # Save config self._save_config() def clean_logs(self): """ Delete all files in the log directory. """ self.logger.info(f"Deleting log files...") logs = self.log_dir.glob('*') for log in logs: os.remove(str(log)) print(f"Deleted: {log}") def clean_all(self): """ Delete all logs and databases and the config. """ self.clean_dbs() self.clean_logs() os.remove(str(self.config_path)) print(f"Deleted: {self.config_path}") os.remove(str(self.config['state file'])) print(f"Deleted: {self.config['state file']}") def _index_local_databases(self) -> dict: need_save = False dbs = copy.deepcopy(self.state['dbs']) db_paths = self.db_dir.glob('*') for db in db_paths: if db.name.endswith('.cdiff') or db.name.endswith('.sign'): # Ignore CDIFFs and sign files, they'll get printed later. continue if db.name not in dbs: version = 0 # Found a file in here that ISN'T a part of the config if db.name.endswith('.cvd'): # Found a CVD in here that ISN'T a part of the config! # Very odd BTW. self.logger.warning(f"Found a CVD in the DB directory that isn't in the config: {db.name}") try: version = self._get_cvd_version_from_file(db) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to determine version for {db.name}") dbs[db.name] = { "url" : "n/a", "retry after" : 0, "last modified" : os.path.getmtime(str(db)), "last checked" : 0, "DNS field" : 0, "local version" : version, "CDIFFs" : [] } else: # DB on disk is from our config if db.name.endswith(".cvd") and self.state['dbs'][db.name]['local version'] == 0: # Seems like we somehow got a (config'd) CVD file in our database directory without # saving the CVD info to the config. Let's just update the version field. self.logger.info(f"Found {db.name} in the DB directory, though it wasn't downloaded using this tool.") try: dbs[db.name]['local version'] = self._get_cvd_version_from_file(self.db_dir / db.name) self.logger.info(f"Identified mysterious {db.name} version: {dbs[db.name]['local version']}") # Add the version info for this mysteriously deposited CVD to our config. self.state['dbs'][db.name]['local version'] = dbs[db.name]['local version'] need_save = True except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to determine version # of mysterious {db.name} file. Perhaps it is corrupted?") if need_save: self._save_config() return dbs def db_list(self) -> None: """ Print list of databases """ dbs = self._index_local_databases() for db in dbs: updated = datetime.datetime.fromtimestamp(dbs[db]['last modified']).strftime('%Y-%m-%d %H:%M:%S') checked = datetime.datetime.fromtimestamp(dbs[db]['last checked']).strftime('%Y-%m-%d %H:%M:%S') self.logger.info(f"Database: {db}") if dbs[db]['last modified'] == 0: self.logger.info(" last modified: not downloaded") else: self.logger.info(f" last modified: {updated}") if dbs[db]['last checked'] == 0: self.logger.debug(" last checked: n/a") else: self.logger.debug(f" last checked: {checked}") self.logger.debug(f" url: {dbs[db]['url']}") if db.endswith(".cvd"): # Only CVD's have versions. self.logger.debug(f" local version: {dbs[db]['local version']}") if len(dbs[db]['CDIFFs']) > 0: self.logger.debug(f" CDIFFs:") for cdiff in dbs[db]['CDIFFs']: self.logger.debug(f" {cdiff}") def db_show(self, name) -> bool: """ Show details for a specific database """ found = False dbs = self._index_local_databases() for db in dbs: if db == name: found = True; updated = datetime.datetime.fromtimestamp(dbs[db]['last modified']).strftime('%Y-%m-%d %H:%M:%S') checked = datetime.datetime.fromtimestamp(dbs[db]['last checked']).strftime('%Y-%m-%d %H:%M:%S') self.logger.info(f"Database: {db}") if dbs[db]['last modified'] == 0: self.logger.info(" last modified: not downloaded") else: self.logger.info(f" last modified: {updated}") if dbs[db]['last checked'] == 0: self.logger.info(" last checked: n/a") else: self.logger.info(f" last checked: {checked}") self.logger.info(f" url: {dbs[db]['url']}") if db.endswith(".cvd"): self.logger.info(f" local version: {dbs[db]['local version']}") if len(dbs[db]['CDIFFs']) > 0: self.logger.info(f" CDIFFs: \n{json.dumps(dbs[db]['CDIFFs'], indent=4)}") return True if not found: self.logger.error(f"No such database: {name}") return found def _query_dns_txt_entry(self) -> bool: ''' Attempt to get version from current.cvd.clamav.net DNS TXT entry ''' got_it = False self.logger.debug(f"Checking available versions via DNS TXT entry query of current.cvd.clamav.net") try: our_resolver = resolver.Resolver() our_resolver.timeout = 5 # Explicitly setting query timeout to mitigate https://github.com/Cisco-Talos/cvdupdate/issues/17 nameservers = self._get_nameserver_configuration() if nameservers: our_resolver.nameservers = nameservers self.logger.info(f"Using nameservers: {nameservers}") else: self.logger.info("Using system configured nameservers") answer = str(our_resolver.resolve("current.cvd.clamav.net","TXT").response.answer[0]) versions = re.search('".*"', answer).group().strip('"') self.dns_version_tokens = versions.split(':') got_it = True except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.warning(f"Failed to determine available version via DNS TXT query!") return got_it def _get_nameserver_configuration(self) -> List[str]: ''' Parse comma delimited nameserver string into a list for Resolver ''' nameserver_string = self._get_nameserver_string() nameservers = [] os_platform = platform.platform() operating_system = os_platform.split("-")[0].lower() if nameserver_string != "": try: nameservers = [x.strip() for x in nameserver_string.split(',')] except Exception as exc: self.logger.warning(f"Failed to parse nameserver configuration: {nameserver_string}, ignoring...") nameservers = [] if nameservers == [] and operating_system == "windows": # Try OpenDNS nameservers on Windows if not overridden by the user, because leaving it empty seems to fail. nameservers = ['208.67.222.222', '208.67.220.220'] return nameservers def _get_nameserver_string(self) -> str: ''' Determine user provided nameserver configuration ''' config_nameserver = self.config['nameserver'] env_nameserver = os.environ.get("CVDUPDATE_NAMESERVER") # Environment variable overrides the configuration setting if env_nameserver != None and env_nameserver != "": self.logger.info(f"Found CVDUPDATE_NAMESERVER environment variable to provide nameservers: {env_nameserver}") return env_nameserver elif config_nameserver != None and config_nameserver != "": self.logger.info(f"Found configuration provided nameservers: {config_nameserver}") return config_nameserver return "" def _query_cvd_version_dns(self, db: str) -> int: ''' This is a faux query. Try to look up the version # from the DNS TXT entry we already have. ''' version = 0 if self.dns_version_tokens == []: # Query DNS if we haven't already for _attempt in range(self.config['max retry']): if self._query_dns_txt_entry(): break # Pause before next attempt. time.sleep(0.1) if self.dns_version_tokens == []: # Query failed. Bail out. return version self.logger.debug(f"Checking {db} version via DNS TXT advertisement.") if self.state['dbs'][db]['DNS field'] == 0: # Invalid DNS field value for database version. self.logger.warning(f"Failed to get DB version from DNS TXT entry: Invalid DNS field value for database version.") return version try: version = int(self.dns_version_tokens[self.state['dbs'][db]['DNS field']]) self.logger.debug(f"{db} version advertised by DNS: {version}") # Update the "last checked" time. self.state['dbs'][db]['last checked'] = time.time() except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.warning(f"Failed to get DB version from DNS TXT entry!") return version def _query_cvd_version_http(self, db: str) -> int: ''' Download the CVD header and read the CVD version Return 1+ if queried version. Return 0 if failed. ''' version = 0 url = self.state['dbs'][db]['url'] self.logger.debug(f"Checking {db} version via HTTP download of CVD header.") ims = datetime.datetime.fromtimestamp(self.state['dbs'][db]['last modified'], tz=datetime.timezone.utc).strftime('%a, %d %b %Y %H:%M:%S GMT') retry = 0 response = None while retry < self.config['max retry']: response = requests.get(url, headers = { 'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})', 'Range': 'bytes=0-95', 'If-Modified-Since': ims, }) if ((response.status_code == 200 or response.status_code == 206) and ('content-length' in response.headers) and (int(response.headers['content-length']) > len(response.content))): self.logger.warning(f"Response was truncated somehow...") self.logger.warning(f" Expected {response.headers['content-length']}") self.logger.warning(f" Received {response.content}, let's retry.") retry += 1 else: break if response is None: self.logger.error(f"No response received requesting CVD header from {url}.") return 0 if response.status_code == 200 or response.status_code == 206: # Looks like we downloaded something... if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)): self.logger.error(f"Failed to download {db} header to check the version #.") return 0 # Successfully downloaded the header. # We used the IMS header so this means it's probably newer, but we'll check just in case. cvd_header = response.content version = self._get_version_from_cvd_header(cvd_header) self.logger.debug(f"{db} version available by HTTP download: {version}") elif response.status_code == 304: # HTTP Not-Modified, it's not newer.than what we already have. # Just return the current local version. version = self.state['dbs'][db]['local version'] self.logger.debug(f"{db} not-modified since: {ims} (local version {version})") elif response.status_code == 429: # Rejected because downloading the same file too frequently. self.logger.warning(f"Failed to download {db} header to check the version #.") self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.") try_again_seconds = 60 * 60 * 12 # 12 hours if 'Retry-After' in response.headers.keys(): try_again_seconds = int(response.headers['Retry-After']) self.state['dbs'][db]['retry after'] = time.time() + float(try_again_seconds) try_again_string = str(datetime.timedelta(seconds=try_again_seconds)) self.logger.warning(f"We won't try {db} again for {try_again_string} hours.") else: # Check failed! self.logger.error(f"Failed to download {db} header to check the version #. Url: {url}") if version > 0: # Update the "last checked" time. self.state['dbs'][db]['last checked'] = time.time() return version def _download_db_from_url(self, db: str, url: str, last_modified: int, version=0) -> CvdStatus: ''' Download contents from a url and save to a filename in the database directory. Will use If-Modified-Since If Not-Modified, it will not replace the current database. ''' ims: str = datetime.datetime.fromtimestamp(last_modified, tz=datetime.timezone.utc).strftime('%a, %d %b %Y %H:%M:%S GMT') retry = 0 response = None while retry < self.config['max retry']: response = requests.get(url, headers = { 'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})', 'If-Modified-Since': ims, }) if ((response.status_code == 200 or response.status_code == 206) and ('content-length' in response.headers) and (int(response.headers['content-length']) > len(response.content))): self.logger.warning(f"Response was truncated somehow...") self.logger.warning(f" Expected {response.headers['content-length']}") self.logger.warning(f" Received {response.content}, let's retry.") retry += 1 else: break if response is None: self.logger.error(f"No response received requesting {url}.") return CvdStatus.ERROR if response.status_code == 200: # Looks like we downloaded something... if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)): self.logger.error(f"Failed to download {db}") return CvdStatus.ERROR # Download Success if version > 0: self.logger.info(f"Downloaded {db}. Version: {version}") else: self.logger.info(f"Downloaded {db}") try: with (self.db_dir / db).open('wb') as new_db: new_db.write(response.content) # Update config w/ new db info self.state['dbs'][db]['last modified'] = time.time() if db.endswith('.cvd'): self.state['dbs'][db]['local version'] = self._get_version_from_cvd_header(response.content[:96]) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to save {db} to {self.db_dir}") return CvdStatus.ERROR elif response.status_code == 304: # Not modified since IMS. We have the latest version. version = self.state['dbs'][db]['local version'] self.logger.info(f"{db} not-modified since: {ims} (local version {version})") # Check for .cvd.sign. # it won't download if we already have it. self._download_sign_file_for( db, url, last_modified=0, version=version) return CvdStatus.NO_UPDATE elif response.status_code == 429: # Rejected because downloading the same file too frequently. self.logger.warning(f"Failed to download {db}.") self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.") try_again_seconds = 60 * 60 * 12 # 12 hours if 'Retry-After' in response.headers.keys(): try_again_seconds = int(response.headers['Retry-After']) self.state['dbs'][db]['retry after'] = time.time() + float(try_again_seconds) try_again_string = str(datetime.timedelta(seconds=try_again_seconds)) self.logger.warning(f"We won't try {db} again for {try_again_string} hours.") # We'll have to retry after the cooldown. return CvdStatus.ERROR else: # HTTP Get failed. self.logger.error(f"Failed to download {db} from {url}") return CvdStatus.ERROR # Now try downloading the corresponding .cvd.sign. # It's okay if it doesn't exist self._download_sign_file_for( db, url, last_modified=0, version=version) return CvdStatus.UPDATED def _download_cdiff(self, db: str, file: str, db_url: str, last_modified: int, desired_version: int, available_version: int) -> CvdStatus: ''' Download a CDIFF file given a file name and version. The file name should be in the format of "daily-12345.cdiff" ''' self.logger.debug(f"Checking for {file}") # now remove the old file name from the db_url and add the new sign file name base_url = db_url.rsplit('/', 1)[0] url = f"{base_url}/{file}" retry = 0 response = None while retry < self.config['max retry']: response = requests.get(url, headers = { 'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})', }) if ((response.status_code == 200 or response.status_code == 206) and ('content-length' in response.headers) and (int(response.headers['content-length']) > len(response.content))): self.logger.warning(f"Response was truncated somehow...") self.logger.warning(f" Expected {response.headers['content-length']}") self.logger.warning(f" Received {response.content}, let's retry.") retry += 1 else: break if response is None: self.logger.error(f"No response received requesting {url}.") return CvdStatus.ERROR if response.status_code == 200: # Looks like we downloaded something... if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)): self.logger.error(f"Failed to download CDIFF.") return CvdStatus.ERROR # Download Success self.logger.info(f"Downloaded {file}") try: with (self.db_dir / f"{file}").open('wb') as new_db: new_db.write(response.content) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to save {file} to {self.db_dir}.") # Update config with CDIFF, for posterity self.state['dbs'][db]['CDIFFs'].append(file) # Prune old CDIFFs if needed if len(self.state['dbs'][db]['CDIFFs']) > self.config['# cdiffs to keep']: try: os.remove(self.db_dir / self.state['dbs'][db]['CDIFFs'][0]) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.debug(f"Tried to prune old cdiffs, but they weren't found, maybe someone else removed them already.") self.state['dbs'][db]['CDIFFs'] = self.state['dbs'][db]['CDIFFs'][1:] elif response.status_code == 429: # Rejected because downloading the same file too frequently. self.logger.warning(f"Failed to download {file}") self.logger.warning(f"Download request rejected because we've downloaded the same file too frequently.") try_again_seconds = 60 * 60 * 12 # 12 hours if 'Retry-After' in response.headers.keys(): try_again_seconds = int(response.headers['Retry-After']) self.state['dbs'][db]['retry after'] = time.time() + float(try_again_seconds) try_again_string = str(datetime.timedelta(seconds=try_again_seconds)) self.logger.warning(f"We won't try {db} again for {try_again_string} hours.") # Sure only a CDIFF failed, but if we want any chance of trying the CDIFF again # in the future, let's bail out now and retry the CVD + CDIFFs after the cooldown. return CvdStatus.ERROR else: # HTTP Get failed. self.logger.info(f"No CDIFF found for {db} version # {desired_version}") if desired_version < available_version: desired_version = available_version - 1 self.logger.info(f"Will just skip to the last CDIFF instead.") else: self.logger.info(f"Giving up on CDIFFs for {db}") return CvdStatus.NO_UPDATE return CvdStatus.UPDATED def _download_sign_file_for(self, file: str, file_url: str, last_modified: int, version=0) -> CvdStatus: ''' Download signature file given a file name. If version > 0, will ensure sign file includes version in the filename, like this: - file-version.ext.sign ''' ims: str = datetime.datetime.fromtimestamp(last_modified, tz=datetime.timezone.utc).strftime('%a, %d %b %Y %H:%M:%S GMT') sign_file = file + ".sign" if version > 0 and str(version) not in file: # the sign file name should include the version in this format: "file-version.ext.sign" # reconstruct. name_parts = file.rsplit('.', 1) if len(name_parts) == 1: self.logger.error(f"Invalid file name. Lacks extension: {file}") return CvdStatus.ERROR file_name = name_parts[0] ext = name_parts[-1] sign_file = f"{file_name}-{version}.{ext}.sign" # check if we already have it. if (self.db_dir / sign_file).exists(): self.logger.debug(f"We already have {sign_file}. Skipping...") return CvdStatus.NO_UPDATE # now remove the old file name from the file_url and add the new sign file name base_url = file_url.rsplit('/', 1)[0] url = f"{base_url}/{sign_file}" retry = 0 response = None while retry < self.config['max retry']: response = requests.get(url, headers = { 'User-Agent': f'CVDUPDATE/{self.version} ({self.state["uuid"]})', 'If-Modified-Since': ims, }) if ((response.status_code == 200 or response.status_code == 206) and ('content-length' in response.headers) and (int(response.headers['content-length']) > len(response.content))): self.logger.warning(f"Response was truncated somehow...") self.logger.warning(f" Expected {response.headers['content-length']}") self.logger.warning(f" Received {response.content}, let's retry.") retry += 1 else: break if response is None: self.logger.error(f"No response received requesting {url}.") return CvdStatus.ERROR if response.status_code == 200: # Looks like we downloaded something... if (('content-length' in response.headers) and int(response.headers['content-length']) > len(response.content)): self.logger.error(f"Failed to download {sign_file}") return CvdStatus.ERROR # Download Success if version > 0: self.logger.info(f"Downloaded {sign_file}. Version: {version}") else: self.logger.info(f"Downloaded {sign_file}") try: with (self.db_dir / sign_file).open('wb') as new_db: new_db.write(response.content) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to save {sign_file} to {self.db_dir}") return CvdStatus.ERROR elif response.status_code == 304: # Not modified since IMS. We have the latest version. self.logger.info(f"{sign_file} not-modified since: {ims} (local version {version})") return CvdStatus.NO_UPDATE elif response.status_code == 429: # Rejected because downloading the same file too frequently. self.logger.warning(f"Failed to download {sign_file} from {url} with 429 response.") return CvdStatus.ERROR else: # HTTP Get failed. self.logger.info(f"Request failed for {url}. Probably no external digital signature provided for {file}") return CvdStatus.ERROR return CvdStatus.UPDATED def _download_cvd(self, db: str, available_version: int) -> CvdStatus: ''' Download the latest available version If we already have some version of the database, attempt to download all CDIFFs in between. If we don't, just get the last two CDIFFs. ''' local_version = self.state['dbs'][db]['local version'] desired_version = local_version + 1 if local_version >= available_version: # Oh! We're already up to date, don't worry about it. self.logger.info(f"{db} is up-to-date. Version: {local_version}") db_url = self.state['dbs'][db]['url'] # Check for the .cvd.sign file, just in case we don't have that yet. # It won't download if we already have it. self._download_sign_file_for( db, db_url, last_modified=0, version=available_version) return CvdStatus.NO_UPDATE elif local_version == 0: # We don't have any version of the DB, let's just get the newest version + the last CDIFF desired_version = available_version # First try to get CDIFFs self.logger.debug(f"Downloading CDIFFs first...") while desired_version <= available_version: # Attempt to download each CDIFF between our local version and the available version. # The url for CVDs should be https://database.clamav.net/ # Eg: # https://database.clamav.net/daily.cvd # For the daily CDIFFs, we would want: # https://database.clamav.net/daily-.cdiff cdiff_file = f"{db[:-len('.cvd')]}-{desired_version}.cdiff" if (self.db_dir / cdiff_file).exists(): self.logger.debug(f"We already have {cdiff_file}. Skipping...") desired_version += 1 continue db_url = self.state['dbs'][db]['url'] # Download the .cdiff result = self._download_cdiff( db, cdiff_file, db_url, last_modified=0, desired_version=desired_version, available_version=available_version) if result != CvdStatus.UPDATED: self.logger.error(f"Failed to download {cdiff_file}.") break # Now try downloading the corresponding .cdiff.sign. # It's okay if it doesn't exist. self._download_sign_file_for( cdiff_file, db_url, last_modified=0, version=desired_version) desired_version += 1 # Now download the available version. desired_version = available_version url = f"{self.state['dbs'][db]['url']}?version={desired_version}" return self._download_db_from_url(db, url, last_modified=0, version=desired_version) def _get_version_from_cvd_header(self, cvd_header: bytes) -> int: ''' Parse a CVD header to read the database version. ''' header_fields = cvd_header.decode('utf-8', 'ignore').strip().split(':') version_found = 0 try: version_found = int(header_fields[2]) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to determine version from CVD header!") return version_found def _get_cvd_version_from_file(self, path: Path) -> int: cvd_header: bytes = b'' version_found = 0 try: with path.open('rb') as cvd_fd: cvd_header = cvd_fd.read(96) if len(cvd_header) < 96: # Most likely a corrupted CVD. Delete. self.logger.debug(f"Failed to read CVD header, perhaps {path.name} is corrupted.") self.logger.debug(f"Will delete {path.name} so it will not cause further problems.") os.remove(str(path)) else: # Got the header, lets parse out the version. version_found = self._get_version_from_cvd_header(cvd_header) except Exception as exc: self.logger.debug(f"EXCEPTION OCCURRED: {exc}") self.logger.error(f"Failed to read version from CVD header from {path}.") if version_found == 0: self.logger.error(f"Failed to determine version from CVD header.") return version_found def pypi_update_check(self): def check(name): """Checks if a newer version of the specified module is available on PyPI.""" self.logger.debug(f'Checking for a newer version of {name}.') try: current_version_str = _get_version(name) current_version = version.parse(current_version_str) response = requests.get(f"https://pypi.org/pypi/{name}/json") # Get package info response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx) latest_version_str = response.json()["info"]["version"] latest_version = version.parse(latest_version_str) if latest_version > current_version: self.logger.warning(f'You are running {name} version: {current_version}.') self.logger.warning(f'There is a newer version on PyPI: {latest_version}. Please update!') return False # Newer version available else: self.logger.debug(f'{name} is up-to-date: {current_version}.') return True # No newer version except requests.exceptions.RequestException as e: self.logger.debug(f"Version check failed: {e}") # Log the error return True # Assume up-to-date on error except PackageNotFoundError: self.logger.error(f"Package {name} not found locally.") return True # Assuming not found means no update possible except Exception as e: # Catch other potential errors (json parsing etc.) self.logger.debug(f"Version check failed: {e}") return True # Assume up-to-date on error return check('cvdupdate') def db_update(self, db="", debug_mode=False) -> int: """ Update one or all of the databases. Returns: Number of errors. """ self.update_errors = 0 self.dbs_updated = 0 self.dns_version_tokens = [] # Make sure we have a database directory to save files to if not self.db_dir.exists(): os.makedirs(self.db_dir) # Check if there is a newer version of CVD-Update self.pypi_update_check() # Query DNS so we can efficiently query CVD version #'s self._query_dns_txt_entry() if self.dns_version_tokens == []: # Query failed. Bail out. self.logger.error(f"Failed to update: DNS query failed.") return 1 if debug_mode: http_client.HTTPConnection.debuglevel = 1 def update(db) -> CvdStatus: ''' Update a database ''' if self.state['dbs'][db]['retry after'] > 0: cooldown_date = datetime.datetime.fromtimestamp(self.state['dbs'][db]['retry after']).strftime('%Y-%m-%d %H:%M:%S') if self.state['dbs'][db]['retry after'] > time.time(): self.logger.warning(f"Skipping {db} which is on cooldown until {cooldown_date}") return CvdStatus.ERROR else: # Cooldown expired. Ok to try again. self.state['dbs'][db]['retry after'] = 0 self.logger.info(f"{db} cooldown expired {cooldown_date}. OK to try again...") if not self.state['dbs'][db]['url'].startswith('http'): self.logger.error(f"Failed to update {db}. Missing or invalid URL: {self.state['dbs'][db]['url']}") return CvdStatus.ERROR self.logger.debug(f"Checking {db} for update from {self.state['dbs'][db]['url']}") if db.endswith('.cvd'): # It's a CVD (official signed clamav database) advertised_version = 0 if (self.db_dir / db).exists(): if self.state['dbs'][db]['local version'] == 0: # Seems like we somehow got a CVD in our database directory without # saving the CVD info to the config. Let's just update the version field. self.state['dbs'][db]['local version'] = self._get_cvd_version_from_file(self.db_dir / db) else: if self.state['dbs'][db]['local version'] != 0: # We have a local version but no CVD in the database directory. # Maybe it was moved or deleted? Let's just reset the local version. self.state['dbs'][db]['local version'] = 0 if self.state['dbs'][db]['DNS field'] > 0: # We can use the DNS TXT fields to check if our version is old. advertised_version = self._query_cvd_version_dns(db) else: # We can't use DNS to see if our version is old. # Use HTTP to pull just the CVD header to check. # First, make sure no one tampered with the DNS field for # main/daily/bytecode when using database.clamav.net if (('database.clamav.net' in self.state['dbs'][db]['url']) and (db == 'main.cvd' or db == 'daily.cvd' or db == 'bytecode.cvd')): self.logger.error(f'It appears that the "DNS field" in {self.config_path} for "{db}" was modified from the default.') self.logger.error(f'Updating {db} from database.clamav.net requires DNS for the version check in order to conserve bandwidth.') self.logger.error(f'Please restore the default settings for the "DNS field" and try again.') return CvdStatus.ERROR advertised_version = self._query_cvd_version_http(db) if advertised_version == 0: self.logger.error(f"Failed to update {db}. Failed to query available CVD version") return CvdStatus.ERROR return self._download_cvd(db, advertised_version) else: # Try the download. # Will use If-Modified-Since # If Not-Modified, it will not replace the current database. return self._download_db_from_url( db, self.state['dbs'][db]['url'], self.state['dbs'][db]['last modified']) if db == "": # Update every DB. for db in self.state['dbs']: status = update(db) if status == CvdStatus.ERROR: self.update_errors += 1 elif status == CvdStatus.UPDATED: self.dbs_updated += 1 else: # Update a specific DB. if db not in self.state['dbs']: self.logger.error(f"Update failed. Unknown database: {db}") else: status = update(db) if status == CvdStatus.ERROR: self.update_errors += 1 elif status == CvdStatus.UPDATED: self.dbs_updated += 1 self._save_config() if self.update_errors == 0 and self.dbs_updated > 0: with (self.db_dir / 'dns.txt').open('w') as dns_file: dns_file.write(':'.join(self.dns_version_tokens)) self.logger.debug(f"Updated {self.db_dir / 'dns.txt'}") return self.update_errors def config_add_db(self, db: str, url: str) -> bool: """ Add another database + url to check when we update. """ extension = db.split('.')[-1] if extension not in [ 'cvd', 'cld', 'cud', 'cfg', 'cat', 'crb', 'ftm', 'ndb', 'ndu', 'ldb', 'ldu', 'idb', 'ydb', 'yar', 'yara', 'cdb', 'cbc', 'pdb', 'gdb', 'wdb', 'hdb', 'hsb', 'hdu', 'hsu', 'mdb', 'msb', 'mdu', 'msu', 'ign', 'ign2', 'info', ]: self.logger.warning(f"{db} does not have valid clamav database file extension.") if db in self.state['dbs']: self.logger.info(f"Cannot add {db}, it is already in our list.") self.logger.info(f"Hint: Try `db list -V` or `db show {db}` for more information.") return False self.state['dbs'][db] = { "url" : url, "retry after" : 0, "last modified" : 0, "last checked" : 0, "DNS field" : 0, "local version" : 0, "CDIFFs" : [] } self.logger.info(f"Added {db} ({url}) to DB list.") self.logger.info(f"{db} will be downloaded next time you run `cvd update` or `cvd update {db}`") self._save_config() return True def config_remove_db(self, db: str) -> bool: """ Remove a database from our list, and delete copies of the DB from the database directory. """ if db not in self.state['dbs']: self.logger.info(f"Cannot remove {db}, it is not in our list.") self.logger.info(f"Hint: Try `db list -V` for more information.") return False try: if (self.db_dir / db).exists(): os.remove(str(self.db_dir / db)) self.logger.info(f"Deleted {db} from database directory.") except Exception as exc: self.logger.debug(f"An exception occured: {exc}") self.logger.error(f"Failed to delete {db} from databse directory!") for cdiff in self.state['dbs'][db]['CDIFFs']: try: if (self.db_dir / cdiff).exists(): os.remove(str(self.db_dir / cdiff)) self.logger.info(f"Deleted {cdiff} from database directory.") except Exception as exc: self.logger.debug(f"An exception occured: {exc}") self.logger.error(f"Failed to delete {cdiff} from databse directory!") self.state['dbs'].pop(db) self.logger.info(f"Removed {db} from DB list.") self._save_config() return True ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1759517099.1814225 cvdupdate-1.2.0/cvdupdate.egg-info/0000755000175100017510000000000015070014653016611 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517099.0 cvdupdate-1.2.0/cvdupdate.egg-info/PKG-INFO0000644000175100017510000003726015070014653017716 0ustar00runnerrunnerMetadata-Version: 2.4 Name: cvdupdate Version: 1.2.0 Summary: ClamAV Private Database Mirror Updater Tool Home-page: https://github.com/Cisco-Talos/cvdupdate Author: The ClamAV Team Author-email: clamav-bugs@external.cisco.com Classifier: Programming Language :: Python :: 3 Classifier: License :: OSI Approved :: Apache Software License Classifier: Operating System :: OS Independent Description-Content-Type: text/markdown License-File: LICENSE Requires-Dist: click>=7.0 Requires-Dist: colorlog>=6.7 Requires-Dist: colorama Requires-Dist: requests Requires-Dist: dnspython>=2.1.0 Requires-Dist: rangehttpserver Requires-Dist: packaging Dynamic: author Dynamic: author-email Dynamic: classifier Dynamic: description Dynamic: description-content-type Dynamic: home-page Dynamic: license-file Dynamic: requires-dist Dynamic: summary

A tool to download and update clamav databases and database patch files for the purposes of hosting your own database mirror.

Copyright (C) 2021-2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.

PyPI version PyPI - Python Version

## About This tool downloads the latest ClamAV databases along with the latest database patch files. This project replaces the `clamdownloader.pl` Perl script by Frederic Vanden Poel, formerly provided here: https://www.clamav.net/documents/private-local-mirrors Run this tool as often as you like, but it will only download new content if there is new content to download. If you somehow manage to download too frequently (eg: by using `cvd clean all` and `cvd update` repeatedly), then the official database server may refuse your download request, and one or more databases may go on cool-down until it's safe to try again. ## Requirements - Python 3.6 or newer. - An internet connection with DNS enabled. - The following Python packages. These will be installed automatically if you use `pip`, but may need to be installed manually otherwise: - `click` v7.0 or newer - `colorlog` v6.7 or newer - `colorama` - `requests` - `dnspython` v2.1.0 or newer - `rangehttpserver` ## Installation You may install `cvdupdate` from PyPI using `pip`, or you may clone the project Git repository and use `pip` to install it locally. Install `cvdupdate` from PyPI: ```bash python3 -m pip install --user cvdupdate ``` ### Updating Your Installation When running `cvd update` to update the databases, it will also check if there is a new version of the `cvdupdate` package on Python's PyPI package repository. If there is a newer version of `cvdupdate`, you will see a message prompting you to upgrade. It will look someething like this: ``` WARNING You are running cvdupdate version: 1.1.0. WARNING There is a newer version on PyPI: 1.1.1. Please update! ``` To upgrade the `cvdupdate` package through PyPI, run: ```bash python3 -m pip install --user --upgrade cvdupdate ``` ## Basic Usage Use the `--help` option with any `cvd` command to get help. ```bash cvd --help ``` > _Tip_: You may not be able to run the `cvd` or `cvdupdate` shortcut directly if your Python Scripts directory is not in your `PATH` environment variable. If you run into this issue, and do not wish to add the Python Scripts directory to your path, you can run CVD-Update like this: > > ```bash > python -m cvdupdate --help > ``` (optional) You may wish to customize where the databases are stored: ```bash cvd config set --dbdir ``` Run this to download the latest database and associated CDIFF patch files: ```bash cvd update ``` Downloaded databases will be placed in `~/.cvdupdate/database` unless you customized it to use a different directory. Newly downloaded databases will replace the previous database version, but the CDIFF patch files will accumulate up to a configured maximum before it starts deleting old CDIFFs (default: 30 CDIFFs). You can configure it to keep more CDIFFs by manually editing the config (default: `~/.cvdupdate/config.json`). The same behavior applies for CVD-Update log rotation. Run this to serve up the database directory on `http://localhost:8000` so you can test it with FreshClam. ```bash cvd serve ``` > _Disclaimer_: The `cvd serve` feature is not intended for production use, just for testing. You probably want to use a more robust HTTP server for production work. Install ClamAV if you don't already have it and, in another terminal window, modify your `freshclam.conf` file. Replace: ``` DatabaseMirror database.clamav.net ``` ... with: ``` DatabaseMirror http://localhost:8000 ``` > _Tip_: A default install on Linux/Unix places `freshclam.conf` in `/usr/local/etc/freshclam.conf`. If one does not exist, you may need to create it using `freshclam.conf.sample` as a template. Now, run `freshclam -v` or `freshclam.exe -v` to see what happens. You should see FreshClam successfully update it's own database directory from your private database server. Run `cvd update` as often as you need. Maybe put it in a `cron` job. > _Tip_: Each command supports a `--verbose` (`-V`) mode, which often provides more details about what's going on under the hood. ### Cron Example Cron is a popular choice to automate frequent tasks on Linux / Unix systems. 1. Open a terminal running as the user which you want CVD-Update to run under, do the following: ```bash crontab -e ``` 2. Press `i` to insert new text, and add this line: ```bash 30 */4 * * * /bin/sh -c "~/.local/bin/cvd update &> /dev/null" ``` Or instead of `~/`, you can do this, replacing `username` with your user name: ```bash 30 */4 * * * /bin/sh -c "/home/username/.local/bin/cvd update &> /dev/null" ``` 3. Press , then type `:wq` and press to write the file to disk and quit. **About these settings**: I selected `30 */4 * * *` to run at minute 30 past every 4th hour. CVD-Update uses a DNS check to do version checks before it attempts to download any files, just like FreshClam. Running CVD-Update more than once a day should not be an issue. CVD-Update will write logs to the `~/.cvdupdate/logs` directory, which is why I directed `stdout` and `stderr` to `/dev/null` instead of a log file. You can use the `cvd config set` command to customize the log directory if you like, or redirect `stdout` and `stderr` to a log file if you prefer everything in one log instead of separate daily logs. ## Optional Functionality ### Using a custom DNS server DNS is required for CVD-Update to function properly (to gather the TXT record containing the current definition database version). You can select a specific nameserver to ensure said nameserver is used when querying the TXT record containing the current database definition version available 1. Set the nameserver in the config. Eg: ```bash cvd config set --nameserver 208.67.222.222 ``` 2. Set the environment variable `CVDUPDATE_NAMESERVER`. Eg: ```bash CVDUPDATE_NAMESERVER="208.67.222.222" cvd update ``` The environment variable will take precedence over the nameserver config setting. Note: Both options can be used to provide a comma-delimited list of nameservers to utilize for resolution. ### Using a proxy Depending on your type of proxy, you may be able to use CVD-Update with your proxy by running CVD-Update like this: First, set a custom domain name server to use the proxy: ```bash cvd config set --nameserver ``` Then run CVD-Update like this: ```bash http_proxy=http://: https_proxy=http://: cvd update -V ``` Or create a script to wrap the CVD-Update call. Something like: ```bash #!/bin/bash http_proxy=http://: export http_proxy https_proxy=http://: export https_proxy cvd update -V ``` > _Disclaimer_: CVD-Update doesn't support proxies that require authentication at this time. If your network admin allows it, you may be able to work around it by updating your proxy to allow HTTP requests through unauthenticated if the User-Agent matches your specific CVD-Update user agent. The CVD-Update User-Agent follows the form `CVDUPDATE/ ()` where the `uuid` is unique to your installation and can be found in the `~/.cvdupdate/state.json` file (or `~/.cvdupdate/config.json` for cvdupdate <=1.0.2). See https://github.com/Cisco-Talos/cvdupdate/issues/9 for more details. > > Adding support for proxy authentication is a ripe opportunity for a community contribution to the project. ## Files and directories created by CVD-Update This tool is to creates the following directories: - `~/.cvdupdate` - `~/.cvdupdate/logs` - `~/.cvdupdate/databases` This tool creates the following files: - `~/.cvdupdate/config.json` - `~/.cvdupdate/state.json` - `~/.cvdupdate/databases/.cvd` - `~/.cvdupdate/databases/-.cdiff` - `~/.cvdupdate/logs/.log` > _Tip_: You can set custom `database` and `logs` directories with the `cvd config set` command. It is likely you will want to customize the `database` directory to point to your HTTP server's `www` directory (or equivalent). Bare in mind that if you already downloaded the databases to the old directory, you may want to move them to the new directory. > _Important_: If you want to use a custom config path, you'll have to use it in every command. If you're fine with having it go in `~/.cvdupdate/config.json`, don't worry about it. ## Additional Usage ### Get familiar with the tool Familiarize yourself with the various commands using the `--help` option. ```bash cvd --help cvd config --help cvd update --help cvd add --help cvd clean --help ``` Print out the current list of databases. ```bash cvd list -V ``` Print out the config to see what it looks like. ```bash cvd config show ``` ### Do an update Do an update, use "verbose mode" to so you can get a feel for how it works. ```bash cvd update -V ``` List out the databases again: ```bash cvd list -V ``` The print out the config again so you can see what's changed. ```bash cvd config show ``` And maybe take a peek in the database directory as well to see it for yourself. ```bash ls ~/.cvdupdate/database ``` Have a look at the logs if you wish. ```bash ls ~/.cvdupdate/logs cat ~/.cvdupdate/logs/* ``` ### Add an additional database Maybe add an additional database that is not part of the default set of databases. ```bash cvd add linux.cvd https://database.clamav.net/linux.cvd ``` List out the databases again: ```bash cvd list -V ``` ### Serve it up, Test out FreshClam Test out your mirror with FreshClam on the same computer. This tool includes a `--serve` feature that will host the current database directory on http://localhost (default port: 8000). You can test it by running `freshclam` or `freshclam.exe` locally, where you've configured `freshclam.conf` with: ``` DatabaseMirror http://localhost:8000 ``` ## Use docker Build docker image ```bash docker build . --tag cvdupdate:latest ``` Run image, that will automaticly update databases in folder `/srv/cvdupdate` and write logs to `/var/log/cvdupdate` ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ cvdupdate:latest ``` Run image, that will automaticly update databases in folder `/srv/cvdupdate`, write logs to `/var/log/cvdupdate` and set owner of files to user with ID 1000 ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ -e USER_ID=1000 \ cvdupdate:latest ``` Default update interval is `30 */4 * * *` (see [Cron Example](#cron-example)) You may pass custom update interval in environment variable `CRON` For example - update every day in 00:00 ```bash docker run -d \ -v /srv/cvdupdate:/cvdupdate/database \ -v /var/log/cvdupdate:/cvdupdate/logs \ -e CRON='0 0 * * *' \ cvdupdate:latest ``` ## Use Docker Compose A Docker `compose.yaml` is provided to: 1. Regularly update a Docker volume with the latest ClamAV databases. 2. Serve a database mirror on port 8000 using the Apache webserver. Edit the `compose.yaml` file if you need to change the default values: * Port 8000 * USER_ID=0 * CRON=30 */4 * * * ### Build ```bash docker compose build ``` ### Start ```bash docker compose up -d ``` ### Stop ```bash docker compose down ``` ### Volumes Volumes are defined in `compose.yaml` and will be auto-created when you run `docker compose up` ``` DRIVER VOLUME NAME local cvdupdate_database local cvdupdate_log ``` ## Contribute We'd love your help. There are many ways to contribute! ### Community Join the ClamAV community on the [ClamAV Discord chat server](https://discord.gg/sGaxA5Q). ### Report issues If you find an issue with CVD-Update or the CVD-Update documentation, please submit an issue to our [GitHub issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Before you submit, please check to if someone else has already reported the issue. ### Development If you find a bug and you're able to craft a fix yourself, consider submitting the fix in a [pull request](https://github.com/Cisco-Talos/cvdupdate/pulls). Your help will be greatly appreciated. If you want to contribute to the project and don't have anything specific in mind, please check out our [issue tracker](https://github.com/Cisco-Talos/cvdupdate/issues). Perhaps you'll be able to fix a bug or add a cool new feature. _By submitting a contribution to the project, you acknowledge and agree to assign Cisco Systems, Inc the copyright for the contribution. If you submit a significant contribution such as a new feature or capability or a large amount of code, you may be asked to sign a contributors license agreement comfirming that Cisco will have copyright license and patent license and that you are authorized to contribute the code._ #### Development Set-up The following steps are intended to help users that wish to contribute to development of the CVD-Update project get started. 1. Create a fork of the [CVD-Update git repository](https://github.com/Cisco-Talos/cvdupdate), and then clone your fork to a local directory. For example: ```bash git clone https://github.com//cvdupdate.git ``` 2. Make sure CVD-Update is not already installed. If it is, remove it. ```bash python3 -m pip uninstall cvdupdate ``` 3. Use pip to install CVD-Update in "edit" mode. ```bash python3 -m pip install -e --user ./cvdupdate ``` Once installed in "edit" mode, any changes you make to your clone of the CVD-Update code will be immediately usable simply by running the `cvdupdate` / `cvd` commands. ### Conduct This project has not selected a specific Code-of-Conduct document at this time. However, contributors are expected to behave in professional and respectful manner. Disrespectful or inappropriate behavior will not be tolerated. ## License CVD-Update is licensed under the Apache License, Version 2.0 (the "License"). You may not use the CVD-Update project except in compliance with the License. A copy of the license is located [here](LICENSE), and is also available online at [apache.org](http://www.apache.org/licenses/LICENSE-2.0). Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517099.0 cvdupdate-1.2.0/cvdupdate.egg-info/SOURCES.txt0000644000175100017510000000057315070014653020502 0ustar00runnerrunnerLICENSE README.md setup.py cvdupdate/__init__.py cvdupdate/__main__.py cvdupdate/auto_updater.py cvdupdate/cvdupdate.py cvdupdate.egg-info/PKG-INFO cvdupdate.egg-info/SOURCES.txt cvdupdate.egg-info/dependency_links.txt cvdupdate.egg-info/entry_points.txt cvdupdate.egg-info/requires.txt cvdupdate.egg-info/top_level.txt tests/__init__.py tests/conftest.py tests/test_cvdupdate.py././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517099.0 cvdupdate-1.2.0/cvdupdate.egg-info/dependency_links.txt0000644000175100017510000000000115070014653022657 0ustar00runnerrunner ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517099.0 cvdupdate-1.2.0/cvdupdate.egg-info/entry_points.txt0000644000175100017510000000012215070014653022102 0ustar00runnerrunner[console_scripts] cvd = cvdupdate.__main__:cli cvdupdate = cvdupdate.__main__:cli ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517099.0 cvdupdate-1.2.0/cvdupdate.egg-info/requires.txt0000644000175100017510000000012615070014653021210 0ustar00runnerrunnerclick>=7.0 colorlog>=6.7 colorama requests dnspython>=2.1.0 rangehttpserver packaging ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517099.0 cvdupdate-1.2.0/cvdupdate.egg-info/top_level.txt0000644000175100017510000000002015070014653021333 0ustar00runnerrunnercvdupdate tests ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1759517099.1824224 cvdupdate-1.2.0/setup.cfg0000644000175100017510000000004615070014653014761 0ustar00runnerrunner[egg_info] tag_build = tag_date = 0 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/setup.py0000644000175100017510000000212615070014643014652 0ustar00runnerrunnerimport setuptools with open("README.md", "r") as fh: long_description = fh.read() setuptools.setup( name="cvdupdate", version="1.2.0", author="The ClamAV Team", author_email="clamav-bugs@external.cisco.com", copyright="Copyright (C) 2025 Cisco Systems, Inc. and/or its affiliates. All rights reserved.", description="ClamAV Private Database Mirror Updater Tool", long_description=long_description, long_description_content_type="text/markdown", url="https://github.com/Cisco-Talos/cvdupdate", packages=setuptools.find_packages(), entry_points={ "console_scripts": [ "cvdupdate = cvdupdate.__main__:cli", "cvd = cvdupdate.__main__:cli", ] }, install_requires=[ "click>=7.0", "colorlog>=6.7", "colorama", "requests", "dnspython>=2.1.0", "rangehttpserver", "packaging", ], classifiers=[ "Programming Language :: Python :: 3", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", ], ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1759517099.1814225 cvdupdate-1.2.0/tests/0000755000175100017510000000000015070014653014302 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/tests/__init__.py0000644000175100017510000000000015070014643016400 0ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/tests/conftest.py0000644000175100017510000000135115070014643016500 0ustar00runnerrunnerfrom textwrap import dedent from glob import glob from pathlib import Path import pytest # grab all files from fixtures/ pytest_plugins = [ f'fixtures.{fi.stem}' for fi in Path(__file__).glob("fixtures/*.py") if fi.stem != '__init__' ] # prevent any test from running if the default .cvdupdate directory already exists @pytest.fixture(scope='session', autouse=True) def fail_if_cvdupdate_dir_exists(): defaultdir = Path.home() / '.cvdupdate' if defaultdir.exists(): pytest.exit(dedent(f''' Error: {defaultdir} exists. Aborting tests to prevent losing actual cvdupdate data. Ensure tests are not running against an actual cvdupdate install. '''), returncode=1 ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1759517091.0 cvdupdate-1.2.0/tests/test_cvdupdate.py0000644000175100017510000001017015070014643017670 0ustar00runnerrunnerimport json from pathlib import Path import shutil from tests.fixtures.revert import revert_homedir from cvdupdate.cvdupdate import CVDUpdate def test_instantiation(revert_homedir): c = CVDUpdate() def test_alternate_config_locations(revert_homedir, tmp_path): ''' Test that we can save config and state files to alternative locations ''' # ensure we're starting with a clean slate default_cvdupdate_dir = Path.home() / '.cvdupdate' assert not default_cvdupdate_dir.exists() # set the config file to be in pytests's /tmp/pytest-* config_file_path = tmp_path / 'config.json' c = CVDUpdate(config=config_file_path) # verify the file is created and has default config data inside assert config_file_path.exists() txt = config_file_path.read_text() assert txt config_file_json = json.loads(txt) assert config_file_json == c.config # state file value will differ, so blank it out for comparing c.config['state file'] = '' assert c.config == c.default_config # verify the state file is created and has default data inside state_file_path = tmp_path / 'state.json' assert state_file_path.exists() txt = state_file_path.read_text() assert txt state_file_json = json.loads(txt) assert state_file_json == c.state # again, uuid will differ, so toss it out del c.state['uuid'] assert c.state == c.default_state # ~/.cvdupdate exists, because we haven't changed the logdir location # but that's all it should have in it default_cvdupdate_dir = Path.home() / '.cvdupdate' assert default_cvdupdate_dir.exists() children = list(default_cvdupdate_dir.iterdir()) assert len(children) == 1 assert children[0] == default_cvdupdate_dir / 'logs' def test_default_config_not_mutated(revert_homedir, tmp_path): ''' default_config and default_state are both class-level attributes Ensure that when we copy these, we are actually copying them and not simply reassigning note that this typically won't be a problem in normal usage, but it was a problem during testing and was really annoying to track down ''' a = CVDUpdate() config_file_path = tmp_path / 'config.json' # set the config file to be in pytests /tmp/pytest-* b = CVDUpdate(config=config_file_path) assert all(val == b.config[key] for key,val in a.config.items() if key != 'state file') assert id(a.config) != id(b.config) assert id(a.default_config) == id(b.default_config) == id(CVDUpdate.default_config) assert a.state != b.state assert id(a.default_state) == id(b.default_state) def test_existing_state_migrates_successfully(revert_homedir): ''' specifically test migrating an existing config.json to config + state.json''' default_cvdupdate_dir = Path.home() / '.cvdupdate' default_cvdupdate_dir.mkdir(parents=True) # create a .cvdupdate/config.json which also contains dbs definitions old_config_file = 'tests/files/v1.0.2.config.json' old_config_json = json.loads(Path(old_config_file).read_text()) old_config_json["log directory"] = str(default_cvdupdate_dir / 'logs') old_config_json["db directory"] = str(default_cvdupdate_dir / 'database') with (default_cvdupdate_dir / 'config.json').open('w') as test_config: json.dump(old_config_json, test_config) # create cvdupdate object, which will read config.json and split state into state.json a = CVDUpdate() new_config_json = old_config_json # create expected state.json contents by copying the bits that move new_state_json = {} new_state_json['dbs'] = old_config_json['dbs'] new_state_json['uuid'] = old_config_json['uuid'] # new update the config.json contents del new_config_json['dbs'] del new_config_json['uuid'] new_config_json['state file'] = str(default_cvdupdate_dir / 'state.json') # compare actual result with expected transform with open(default_cvdupdate_dir / 'config.json') as config: assert new_config_json == json.loads(config.read()) with open(default_cvdupdate_dir / 'state.json') as state: from pprint import pprint assert new_state_json == json.loads(state.read())