zopfli-0.8.1/.cargo_vcs_info.json 0000644 00000000136 00000000001 0012345 0 ustar {
"git": {
"sha1": "95946a0e94e88a882b3eb80d2eaa576904096d96"
},
"path_in_vcs": ""
} zopfli-0.8.1/CONTRIBUTORS 0000644 0000000 0000000 00000000211 10461020230 0013007 0 ustar 0000000 0000000 Mark Adler
Jyrki Alakuijala
Frédéric Kayser
Jeffrey Lim
Daniel Reed
Huzaifa Sidhpurwala
Péter Szabó
Lode Vandevenne
Derek Buitenhuis
zopfli-0.8.1/COPYING 0000644 0000000 0000000 00000026115 10461020230 0012175 0 ustar 0000000 0000000 Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2011 Google Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
zopfli-0.8.1/Cargo.lock 0000644 00000026440 00000000001 0010326 0 ustar # This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "adler"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe"
[[package]]
name = "autocfg"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0c4b4d0bd25bd0b74681c0ad21497610ce1b7c91b1022cd21c80c6fbdd9476b0"
[[package]]
name = "bit-set"
version = "0.5.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0700ddab506f33b20a03b13996eccd309a48e5ff77d0d95926aa0210fb4e95f1"
dependencies = [
"bit-vec",
]
[[package]]
name = "bit-vec"
version = "0.6.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "349f9b6a179ed607305526ca489b34ad0a41aed5f7980fa90eb03160b69598fb"
[[package]]
name = "bitflags"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf4b9d6a944f767f8e5e0db018570623c85f3d925ac718db4e06d0187adb21c1"
[[package]]
name = "bumpalo"
version = "3.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "79296716171880943b8470b5f8d03aa55eb2e645a4874bdbb28adb49162e012c"
[[package]]
name = "cfg-if"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
[[package]]
name = "crc32fast"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b3855a8a784b474f333699ef2bbca9db2c4a1f6d9088a90a2d25b1eb53111eaa"
dependencies = [
"cfg-if",
]
[[package]]
name = "errno"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "534c5cf6194dfab3db3242765c03bbe257cf92f22b38f6bc0c58d59108a820ba"
dependencies = [
"libc",
"windows-sys",
]
[[package]]
name = "fastrand"
version = "2.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9fc0510504f03c51ada170672ac806f1f105a88aa97a5281117e1ddc3368e51a"
[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "getrandom"
version = "0.2.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7"
dependencies = [
"cfg-if",
"libc",
"wasi",
]
[[package]]
name = "lazy_static"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646"
[[package]]
name = "libc"
version = "0.2.155"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "97b3888a4aecf77e811145cadf6eef5901f4782c53886191b2f693f24761847c"
[[package]]
name = "libm"
version = "0.2.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4ec2a862134d2a7d32d7983ddcdd1c4923530833c9f2ea1a44fc5fa473989058"
[[package]]
name = "linux-raw-sys"
version = "0.4.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "78b3ae25bc7c8c38cec158d1f2757ee79e9b3740fbc7ccf0e59e4b08d793fa89"
[[package]]
name = "lockfree-object-pool"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a69c0481fc2424cb55795de7da41add33372ea75a94f9b6588ab6a2826dfebc"
[[package]]
name = "log"
version = "0.4.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90ed8c1e510134f979dbc4f070f87d4313098b704861a105fe34231c70a3901c"
[[package]]
name = "miniz_oxide"
version = "0.7.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87dfd01fe195c66b572b37921ad8803d010623c0aca821bea2302239d155cdae"
dependencies = [
"adler",
]
[[package]]
name = "num-traits"
version = "0.2.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841"
dependencies = [
"autocfg",
"libm",
]
[[package]]
name = "once_cell"
version = "1.19.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
[[package]]
name = "ppv-lite86"
version = "0.2.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de"
[[package]]
name = "proc-macro2"
version = "1.0.82"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ad3d49ab951a01fbaafe34f2ec74122942fe18a3f9814c3268f1bb72042131b"
dependencies = [
"unicode-ident",
]
[[package]]
name = "proptest"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "31b476131c3c86cb68032fdc5cb6d5a1045e3e42d96b69fa599fd77701e1f5bf"
dependencies = [
"bit-set",
"bit-vec",
"bitflags",
"lazy_static",
"num-traits",
"rand",
"rand_chacha",
"rand_xorshift",
"regex-syntax",
"rusty-fork",
"tempfile",
"unarray",
]
[[package]]
name = "proptest-derive"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9cf16337405ca084e9c78985114633b6827711d22b9e6ef6c6c0d665eb3f0b6e"
dependencies = [
"proc-macro2",
"quote",
"syn",
]
[[package]]
name = "quick-error"
version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quote"
version = "1.0.36"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fa76aaf39101c457836aec0ce2316dbdc3ab723cdda1c6bd4e6ad4208acaca7"
dependencies = [
"proc-macro2",
]
[[package]]
name = "rand"
version = "0.8.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
dependencies = [
"libc",
"rand_chacha",
"rand_core",
]
[[package]]
name = "rand_chacha"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
dependencies = [
"ppv-lite86",
"rand_core",
]
[[package]]
name = "rand_core"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
dependencies = [
"getrandom",
]
[[package]]
name = "rand_xorshift"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d25bf25ec5ae4a3f1b92f929810509a2f53d7dca2f50b794ff57e3face536c8f"
dependencies = [
"rand_core",
]
[[package]]
name = "regex-syntax"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "adad44e29e4c806119491a7f06f03de4d1af22c3a680dd47f1e6e179439d1f56"
[[package]]
name = "rustix"
version = "0.38.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "70dc5ec042f7a43c4a73241207cecc9873a06d45debb38b329f8541d85c2730f"
dependencies = [
"bitflags",
"errno",
"libc",
"linux-raw-sys",
"windows-sys",
]
[[package]]
name = "rusty-fork"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb3dcc6e454c328bb824492db107ab7c0ae8fcffe4ad210136ef014458c1bc4f"
dependencies = [
"fnv",
"quick-error",
"tempfile",
"wait-timeout",
]
[[package]]
name = "simd-adler32"
version = "0.3.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d66dc143e6b11c1eddc06d5c423cfc97062865baf299914ab64caa38182078fe"
[[package]]
name = "syn"
version = "1.0.109"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "tempfile"
version = "3.10.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "85b77fafb263dd9d05cbeac119526425676db3784113aa9295c88498cbf8bff1"
dependencies = [
"cfg-if",
"fastrand",
"rustix",
"windows-sys",
]
[[package]]
name = "unarray"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94"
[[package]]
name = "unicode-ident"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
[[package]]
name = "wait-timeout"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f200f5b12eb75f8c1ed65abd4b2db8a6e1b138a20de009dacee265a2498f3f6"
dependencies = [
"libc",
]
[[package]]
name = "wasi"
version = "0.11.0+wasi-snapshot-preview1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
[[package]]
name = "windows-sys"
version = "0.52.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d"
dependencies = [
"windows-targets",
]
[[package]]
name = "windows-targets"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6f0713a46559409d202e70e28227288446bf7841d3211583a4b53e3f6d96e7eb"
dependencies = [
"windows_aarch64_gnullvm",
"windows_aarch64_msvc",
"windows_i686_gnu",
"windows_i686_gnullvm",
"windows_i686_msvc",
"windows_x86_64_gnu",
"windows_x86_64_gnullvm",
"windows_x86_64_msvc",
]
[[package]]
name = "windows_aarch64_gnullvm"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7088eed71e8b8dda258ecc8bac5fb1153c5cffaf2578fc8ff5d61e23578d3263"
[[package]]
name = "windows_aarch64_msvc"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9985fd1504e250c615ca5f281c3f7a6da76213ebd5ccc9561496568a2752afb6"
[[package]]
name = "windows_i686_gnu"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "88ba073cf16d5372720ec942a8ccbf61626074c6d4dd2e745299726ce8b89670"
[[package]]
name = "windows_i686_gnullvm"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "87f4261229030a858f36b459e748ae97545d6f1ec60e5e0d6a3d32e0dc232ee9"
[[package]]
name = "windows_i686_msvc"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db3c2bf3d13d5b658be73463284eaf12830ac9a26a90c717b7f771dfe97487bf"
[[package]]
name = "windows_x86_64_gnu"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4e4246f76bdeff09eb48875a0fd3e2af6aada79d409d33011886d3e1581517d9"
[[package]]
name = "windows_x86_64_gnullvm"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "852298e482cd67c356ddd9570386e2862b5673c85bd5f88df9ab6802b334c596"
[[package]]
name = "windows_x86_64_msvc"
version = "0.52.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bec47e5bfd1bff0eeaf6d8b485cc1074891a197ab4225d504cb7a1ab88b02bf0"
[[package]]
name = "zopfli"
version = "0.8.1"
dependencies = [
"bumpalo",
"crc32fast",
"lockfree-object-pool",
"log",
"miniz_oxide",
"once_cell",
"proptest",
"proptest-derive",
"simd-adler32",
]
zopfli-0.8.1/Cargo.toml 0000644 00000004116 00000000001 0010345 0 ustar # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO
#
# When uploading crates to the registry Cargo will automatically
# "normalize" Cargo.toml files for maximal compatibility
# with all versions of Cargo and also rewrite `path` dependencies
# to registry (e.g., crates.io) dependencies.
#
# If you are reading this file be aware that the original Cargo.toml
# will likely look very different (and much more reasonable).
# See Cargo.toml.orig for the original contents.
[package]
edition = "2021"
rust-version = "1.73.0"
name = "zopfli"
version = "0.8.1"
build = false
exclude = [
".github/*",
".gitignore",
"Makefile",
"benchmark-builds/*",
"rustfmt.toml",
"test/*",
]
autobins = false
autoexamples = false
autotests = false
autobenches = false
description = "A Rust implementation of the Zopfli compression algorithm."
homepage = "https://github.com/zopfli-rs/zopfli"
readme = "README.md"
keywords = ["compression"]
categories = [
"compression",
"no-std",
]
license = "Apache-2.0"
repository = "https://github.com/zopfli-rs/zopfli"
[package.metadata.docs.rs]
cargo-args = ["--all-features"]
[profile.release]
debug = 2
[lib]
name = "zopfli"
path = "src/lib.rs"
[[bin]]
name = "zopfli"
path = "src/main.rs"
required-features = [
"std",
"gzip",
"zlib",
]
[dependencies.bumpalo]
version = "3.16.0"
[dependencies.crc32fast]
version = "1.4.0"
optional = true
default-features = false
[dependencies.lockfree-object-pool]
version = "0.1.5"
optional = true
[dependencies.log]
version = "0.4.21"
optional = true
[dependencies.once_cell]
version = "1.19.0"
optional = true
[dependencies.simd-adler32]
version = "0.3.7"
optional = true
default-features = false
[dev-dependencies.miniz_oxide]
version = "0.7.3"
[dev-dependencies.proptest]
version = "1.4.0"
[dev-dependencies.proptest-derive]
version = "0.4.0"
[features]
default = [
"std",
"gzip",
"zlib",
]
gzip = ["dep:crc32fast"]
nightly = ["crc32fast?/nightly"]
std = [
"dep:log",
"dep:lockfree-object-pool",
"dep:once_cell",
"crc32fast?/std",
"simd-adler32?/std",
]
zlib = ["dep:simd-adler32"]
zopfli-0.8.1/Cargo.toml.orig 0000644 0000000 0000000 00000003103 10461020230 0014021 0 ustar 0000000 0000000 # The binary target must be set in a single line like this to make it easily
# removable by the CI "Check effective no_std compatibility" step
bin = [{ name = "zopfli", required-features = ["std", "gzip", "zlib"] }]
[package]
name = "zopfli"
version = "0.8.1"
description = "A Rust implementation of the Zopfli compression algorithm."
license = "Apache-2.0"
keywords = ["compression"]
homepage = "https://github.com/zopfli-rs/zopfli"
repository = "https://github.com/zopfli-rs/zopfli"
readme = "README.md"
categories = ["compression", "no-std"]
exclude = [
".github/*",
".gitignore",
"Makefile",
"benchmark-builds/*",
"rustfmt.toml",
"test/*",
]
edition = "2021"
rust-version = "1.73.0"
[dependencies]
crc32fast = { version = "1.4.0", default-features = false, optional = true }
simd-adler32 = { version = "0.3.7", default-features = false, optional = true }
bumpalo = "3.16.0"
log = { version = "0.4.21", optional = true }
lockfree-object-pool = { version = "0.1.5", optional = true }
once_cell = { version = "1.19.0", optional = true }
[dev-dependencies]
proptest = "1.4.0"
proptest-derive = "0.4.0"
miniz_oxide = "0.7.3"
[features]
default = ["std", "gzip", "zlib"]
gzip = ["dep:crc32fast"]
zlib = ["dep:simd-adler32"]
std = [
"dep:log",
"dep:lockfree-object-pool",
"dep:once_cell",
"crc32fast?/std",
"simd-adler32?/std",
]
nightly = ["crc32fast?/nightly"]
[profile.release]
debug = true
# docs.rs uses a nightly toolchain, so it can leverage unstable rustdoc features.
# Reference: https://docs.rs/about/builds
[package.metadata.docs.rs]
cargo-args = ["--all-features"]
zopfli-0.8.1/README.md 0000644 0000000 0000000 00000004276 10461020230 0012425 0 ustar 0000000 0000000
This is a reimplementation of the [Zopfli](https://github.com/google/zopfli) compression tool in Rust.
Carol Nichols started the Rust implementation as an experiment in incrementally rewriting a C library in Rust, keeping the project compiling at every step. For more information about that experiment, see [the slides for a talk she gave about it](https://github.com/carols10cents/rust-out-your-c-talk) and [the repo as it was for the experiment](https://github.com/carols10cents/zopfli).
The minimum supported Rust version (MSRV) for this crate is 1.73. Bumping this version is not considered a breaking change for semantic versioning purposes. We will try to do it only when we estimate that such a bump would not cause widespread inconvenience or breakage.
## How to build
To build the code, run:
```
$ cargo build --release
```
and the executable will be in `target/release/zopfli`.
This should work on stable or beta Rust.
You can also run `make zopfli`, which will run `cargo build` and then symlink `target/release/zopfli` to just `zopfli` in the project root; this is what the C library does and it was useful for scripting purposes during the rewrite process to keep the command and resulting artifacts the same.
## Running the tests
There are some unit tests, mostly around the boundary package merge algorithm implementation in katajainen.rs, and a property-based test for compression reversibility. These tests can be run with:
```
$ cargo test
```
Golden master tests, to check that compressed files are exactly the same as the C implementation would generate, can be run using:
```
$ ./test/run.sh
```
and then checking that git reports no changes to the files in `test/results`.
Or you can run `make test`, which will run `cargo test` followed by `./test/run.sh`; it will fail if there are any mismatches.
zopfli-0.8.1/src/blocksplitter.rs 0000644 0000000 0000000 00000017712 10461020230 0015163 0 ustar 0000000 0000000 use alloc::vec::Vec;
#[cfg(feature = "std")]
use log::{debug, log_enabled};
use crate::{cache::NoCache, deflate::calculate_block_size_auto_type, lz77::Lz77Store};
/// Finds minimum of function `f(i)` where `i` is of type `usize`, `f(i)` is of type
/// `f64`, `i` is in range `start-end` (excluding `end`).
/// Returns the index to the minimum and the minimum value.
fn find_minimum f64>(f: F, start: usize, end: usize) -> (usize, f64) {
if end - start < 1024 {
let mut best = f64::INFINITY;
let mut result = start;
for i in start..end {
let v = f(i);
if v < best {
best = v;
result = i;
}
}
(result, best)
} else {
/* Try to find minimum faster by recursively checking multiple points. */
let mut start = start;
let mut end = end;
const NUM: usize = 9; /* Good value: 9. ?!?!?!?! */
let mut p = [0; NUM];
let mut vp = [0.0; NUM];
let mut lastbest = f64::INFINITY;
let mut pos = start;
while end - start > NUM {
let mut besti = 0;
let mut best = f64::INFINITY;
let multiplier = (end - start) / (NUM + 1);
for i in 0..NUM {
p[i] = start + (i + 1) * multiplier;
vp[i] = f(p[i]);
if vp[i] < best {
best = vp[i];
besti = i;
}
}
if best > lastbest {
break;
}
start = if besti == 0 { start } else { p[besti - 1] };
end = if besti == NUM - 1 { end } else { p[besti + 1] };
pos = p[besti];
lastbest = best;
}
(pos, lastbest)
}
}
/// Returns estimated cost of a block in bits. It includes the size to encode the
/// tree and the size to encode all literal, length and distance symbols and their
/// extra bits.
///
/// litlens: lz77 lit/lengths
/// dists: ll77 distances
/// lstart: start of block
/// lend: end of block (not inclusive)
fn estimate_cost(lz77: &Lz77Store, lstart: usize, lend: usize) -> f64 {
calculate_block_size_auto_type(lz77, lstart, lend)
}
/// Finds next block to try to split, the largest of the available ones.
/// The largest is chosen to make sure that if only a limited amount of blocks is
/// requested, their sizes are spread evenly.
/// lz77size: the size of the LL77 data, which is the size of the done array here.
/// done: array indicating which blocks starting at that position are no longer
/// splittable (splitting them increases rather than decreases cost).
/// splitpoints: the splitpoints found so far.
/// npoints: the amount of splitpoints found so far.
/// lstart: output variable, giving start of block.
/// lend: output variable, giving end of block.
/// returns 1 if a block was found, 0 if no block found (all are done).
fn find_largest_splittable_block(
lz77size: usize,
done: &[u8],
splitpoints: &[usize],
) -> Option<(usize, usize)> {
let mut longest = 0;
let mut found = None;
let mut last = 0;
for &item in splitpoints.iter() {
if done[last] == 0 && item - last > longest {
found = Some((last, item));
longest = item - last;
}
last = item;
}
let end = lz77size - 1;
if done[last] == 0 && end - last > longest {
found = Some((last, end));
}
found
}
/// Prints the block split points as decimal and hex values in the terminal.
#[inline]
fn print_block_split_points(lz77: &Lz77Store, lz77splitpoints: &[usize]) {
if !log_enabled!(log::Level::Debug) {
return;
}
let nlz77points = lz77splitpoints.len();
let mut splitpoints = Vec::with_capacity(nlz77points);
/* The input is given as lz77 indices, but we want to see the uncompressed
index values. */
let mut pos = 0;
if nlz77points > 0 {
for (i, item) in lz77.litlens.iter().enumerate() {
let length = item.size();
if lz77splitpoints[splitpoints.len()] == i {
splitpoints.push(pos);
if splitpoints.len() == nlz77points {
break;
}
}
pos += length;
}
}
debug_assert_eq!(splitpoints.len(), nlz77points);
debug!(
"block split points: {} (hex: {})",
splitpoints
.iter()
.map(|&sp| format!("{}", sp))
.collect::>()
.join(" "),
splitpoints
.iter()
.map(|&sp| format!("{:x}", sp))
.collect::>()
.join(" ")
);
}
/// Does blocksplitting on LZ77 data.
/// The output splitpoints are indices in the LZ77 data.
/// maxblocks: set a limit to the amount of blocks. Set to 0 to mean no limit.
pub fn blocksplit_lz77(lz77: &Lz77Store, maxblocks: u16, splitpoints: &mut Vec) {
if lz77.size() < 10 {
return; /* This code fails on tiny files. */
}
let mut numblocks = 1u32;
let mut done = vec![0; lz77.size()];
let mut lstart = 0;
let mut lend = lz77.size();
while maxblocks != 0 && numblocks < maxblocks as u32 {
debug_assert!(lstart < lend);
let find_minimum_result = find_minimum(
|i| estimate_cost(lz77, lstart, i) + estimate_cost(lz77, i, lend),
lstart + 1,
lend,
);
let llpos = find_minimum_result.0;
let splitcost = find_minimum_result.1;
debug_assert!(llpos > lstart);
debug_assert!(llpos < lend);
let origcost = estimate_cost(lz77, lstart, lend);
if splitcost > origcost || llpos == lstart + 1 || llpos == lend {
done[lstart] = 1;
} else {
splitpoints.push(llpos);
splitpoints.sort();
numblocks += 1;
}
// If `find_largest_splittable_block` returns `None`, no further split will
// likely reduce compression.
let is_finished = find_largest_splittable_block(lz77.size(), &done, splitpoints).map_or(
true,
|(start, end)| {
lstart = start;
lend = end;
lend - lstart < 10
},
);
if is_finished {
break;
}
}
print_block_split_points(lz77, splitpoints);
}
/// Does blocksplitting on uncompressed data.
/// The output splitpoints are indices in the uncompressed bytes.
///
/// options: general program options.
/// in: uncompressed input data
/// instart: where to start splitting
/// inend: where to end splitting (not inclusive)
/// maxblocks: maximum amount of blocks to split into, or 0 for no limit
/// splitpoints: dynamic array to put the resulting split point coordinates into.
/// The coordinates are indices in the input array.
/// npoints: pointer to amount of splitpoints, for the dynamic array. The amount of
/// blocks is the amount of splitpoitns + 1.
pub fn blocksplit(
in_data: &[u8],
instart: usize,
inend: usize,
maxblocks: u16,
splitpoints: &mut Vec,
) {
splitpoints.clear();
let mut store = Lz77Store::new();
/* Unintuitively, Using a simple LZ77 method here instead of lz77_optimal
results in better blocks. */
{
store.greedy(&mut NoCache, in_data, instart, inend);
}
let mut lz77splitpoints = Vec::with_capacity(maxblocks as usize);
blocksplit_lz77(&store, maxblocks, &mut lz77splitpoints);
let nlz77points = lz77splitpoints.len();
/* Convert LZ77 positions to positions in the uncompressed input. */
let mut pos = instart;
if nlz77points > 0 {
for (i, item) in store.litlens.iter().enumerate() {
let length = item.size();
if lz77splitpoints[splitpoints.len()] == i {
splitpoints.push(pos);
if splitpoints.len() == nlz77points {
break;
}
}
pos += length;
}
}
debug_assert_eq!(splitpoints.len(), nlz77points);
}
zopfli-0.8.1/src/cache.rs 0000644 0000000 0000000 00000017225 10461020230 0013344 0 ustar 0000000 0000000 use alloc::vec::Vec;
use core::cmp;
use crate::{
lz77::LongestMatch,
util::{ZOPFLI_CACHE_LENGTH, ZOPFLI_MAX_MATCH, ZOPFLI_MIN_MATCH},
};
// Cache used by ZopfliFindLongestMatch to remember previously found length/dist
// values.
// This is needed because the squeeze runs will ask these values multiple times for
// the same position.
// Uses large amounts of memory, since it has to remember the distance belonging
// to every possible shorter-than-the-best length (the so called "sublen" array).
pub struct ZopfliLongestMatchCache {
length: Vec,
dist: Vec,
sublen: Vec,
}
impl ZopfliLongestMatchCache {
pub fn new(blocksize: usize) -> ZopfliLongestMatchCache {
ZopfliLongestMatchCache {
/* length > 0 and dist 0 is invalid combination, which indicates on purpose
that this cache value is not filled in yet. */
length: vec![1; blocksize],
dist: vec![0; blocksize],
/* Rather large amount of memory. */
sublen: vec![0; ZOPFLI_CACHE_LENGTH * blocksize * 3],
}
}
fn length_at(&self, pos: usize) -> u16 {
self.length[pos]
}
fn dist_at(&self, pos: usize) -> u16 {
self.dist[pos]
}
fn store_length_at(&mut self, pos: usize, val: u16) {
self.length[pos] = val;
}
fn store_dist_at(&mut self, pos: usize, val: u16) {
self.dist[pos] = val;
}
/// Returns the length up to which could be stored in the cache.
fn max_sublen(&self, pos: usize) -> u32 {
let start = ZOPFLI_CACHE_LENGTH * pos * 3;
if self.sublen[start + 1] == 0 && self.sublen[start + 2] == 0 {
return 0; // No sublen cached.
}
self.sublen[start + ((ZOPFLI_CACHE_LENGTH - 1) * 3)] as u32 + 3
}
/// Stores sublen array in the cache.
fn store_sublen(&mut self, sublen: &[u16], pos: usize, length: usize) {
if length < 3 {
return;
}
let start = ZOPFLI_CACHE_LENGTH * pos * 3;
let mut i = 3;
let mut j = 0;
let mut bestlength = 0;
while i <= length {
if i == length || sublen[i] != sublen[i + 1] {
self.sublen[start + (j * 3)] = (i - 3) as u8;
self.sublen[start + (j * 3 + 1)] = sublen[i].wrapping_rem(256) as u8;
self.sublen[start + (j * 3 + 2)] = (sublen[i] >> 8).wrapping_rem(256) as u8;
bestlength = i as u32;
j += 1;
if j >= ZOPFLI_CACHE_LENGTH {
break;
}
}
i += 1;
}
if j < ZOPFLI_CACHE_LENGTH {
debug_assert_eq!(bestlength, length as u32);
self.sublen[start + ((ZOPFLI_CACHE_LENGTH - 1) * 3)] = (bestlength - 3) as u8;
} else {
debug_assert!(bestlength <= length as u32);
}
debug_assert_eq!(bestlength, self.max_sublen(pos));
}
/// Extracts sublen array from the cache.
fn fetch_sublen(&self, pos: usize, length: usize, sublen: &mut [u16]) {
if length < 3 {
return;
}
let start = ZOPFLI_CACHE_LENGTH * pos * 3;
let maxlength = self.max_sublen(pos) as usize;
let mut prevlength = 0;
for j in 0..ZOPFLI_CACHE_LENGTH {
let length = self.sublen[start + (j * 3)] as usize + 3;
let dist = self.sublen[start + (j * 3 + 1)] as u16
+ 256 * self.sublen[start + (j * 3 + 2)] as u16;
let mut i = prevlength;
while i <= length {
sublen[i] = dist;
i += 1;
}
if length == maxlength {
break;
}
prevlength = length + 1;
}
}
}
pub trait Cache {
fn try_get(
&self,
pos: usize,
limit: usize,
sublen: &mut Option<&mut [u16]>,
blockstart: usize,
) -> LongestMatch;
fn store(
&mut self,
pos: usize,
limit: usize,
sublen: &mut Option<&mut [u16]>,
distance: u16,
length: u16,
blockstart: usize,
);
}
pub struct NoCache;
impl Cache for NoCache {
fn try_get(
&self,
_: usize,
limit: usize,
_: &mut Option<&mut [u16]>,
_: usize,
) -> LongestMatch {
LongestMatch::new(limit)
}
fn store(&mut self, _: usize, _: usize, _: &mut Option<&mut [u16]>, _: u16, _: u16, _: usize) {
// Nowhere to store
}
}
impl Cache for ZopfliLongestMatchCache {
fn try_get(
&self,
pos: usize,
mut limit: usize,
sublen: &mut Option<&mut [u16]>,
blockstart: usize,
) -> LongestMatch {
let mut longest_match = LongestMatch::new(limit);
/* The LMC cache starts at the beginning of the block rather than the
beginning of the whole array. */
let lmcpos = pos - blockstart;
/* Length > 0 and dist 0 is invalid combination, which indicates on purpose
that this cache value is not filled in yet. */
let length_lmcpos = self.length_at(lmcpos);
let dist_lmcpos = self.dist_at(lmcpos);
let cache_available = length_lmcpos == 0 || dist_lmcpos != 0;
let max_sublen = self.max_sublen(lmcpos);
let limit_ok_for_cache = limit == ZOPFLI_MAX_MATCH
|| length_lmcpos <= limit as u16
|| (sublen.is_some() && max_sublen >= limit as u32);
if limit_ok_for_cache && cache_available {
if sublen.is_none() || length_lmcpos as u32 <= max_sublen {
let length = cmp::min(length_lmcpos, limit as u16);
let distance;
if let Some(ref mut subl) = *sublen {
self.fetch_sublen(lmcpos, length as usize, subl);
distance = subl[length as usize];
if limit == ZOPFLI_MAX_MATCH && length >= ZOPFLI_MIN_MATCH as u16 {
debug_assert_eq!(subl[length as usize], dist_lmcpos);
}
} else {
distance = dist_lmcpos;
}
longest_match.distance = distance;
longest_match.length = length;
longest_match.from_cache = true;
return longest_match;
}
/* Can't use much of the cache, since the "sublens" need to be calculated,
but at least we already know when to stop. */
limit = length_lmcpos as usize;
longest_match.limit = limit;
}
longest_match
}
fn store(
&mut self,
pos: usize,
limit: usize,
sublen: &mut Option<&mut [u16]>,
distance: u16,
length: u16,
blockstart: usize,
) {
if let Some(ref mut subl) = *sublen {
/* Length > 0 and dist 0 is invalid combination, which indicates on purpose
that this cache value is not filled in yet. */
let lmcpos = pos - blockstart;
let cache_available = self.length_at(lmcpos) == 0 || self.dist_at(lmcpos) != 0;
if limit == ZOPFLI_MAX_MATCH && !cache_available {
debug_assert!(self.length_at(lmcpos) == 1 && self.dist_at(lmcpos) == 0);
if length < ZOPFLI_MIN_MATCH as u16 {
self.store_dist_at(lmcpos, 0);
self.store_length_at(lmcpos, 0);
} else {
self.store_dist_at(lmcpos, distance);
self.store_length_at(lmcpos, length);
}
debug_assert!(!(self.length_at(lmcpos) == 1 && self.dist_at(lmcpos) == 0));
self.store_sublen(subl, lmcpos, length as usize);
}
}
}
}
zopfli-0.8.1/src/deflate.rs 0000644 0000000 0000000 00000132277 10461020230 0013712 0 ustar 0000000 0000000 use alloc::vec::Vec;
use core::{cmp, iter};
#[cfg(feature = "std")]
use log::{debug, log_enabled};
use crate::{
blocksplitter::{blocksplit, blocksplit_lz77},
cache::ZopfliLongestMatchCache,
iter::ToFlagLastIterator,
katajainen::length_limited_code_lengths,
lz77::{LitLen, Lz77Store},
squeeze::{lz77_optimal, lz77_optimal_fixed},
symbols::{
get_dist_extra_bits, get_dist_extra_bits_value, get_dist_symbol,
get_dist_symbol_extra_bits, get_length_extra_bits, get_length_extra_bits_value,
get_length_symbol, get_length_symbol_extra_bits,
},
tree::lengths_to_symbols,
util::{ZOPFLI_NUM_D, ZOPFLI_NUM_LL, ZOPFLI_WINDOW_SIZE},
Error, Options, Write,
};
/// A DEFLATE encoder powered by the Zopfli algorithm that compresses data written
/// to it to the specified sink. Most users will find using [`compress`](crate::compress)
/// easier and more performant.
///
/// The data will be compressed as soon as possible, without trying to fill a
/// backreference window. As a consequence, frequent short writes may cause more
/// DEFLATE blocks to be emitted with less optimal Huffman trees, which can hurt
/// compression and runtime. If they are a concern, short writes can be conveniently
/// dealt with by wrapping this encoder with a [`BufWriter`](std::io::BufWriter), as done
/// by the [`new_buffered`](DeflateEncoder::new_buffered) method. An adequate write size
/// would be >32 KiB, which allows the second complete chunk to leverage a full-sized
/// backreference window.
pub struct DeflateEncoder {
options: Options,
btype: BlockType,
have_chunk: bool,
chunk_start: usize,
window_and_chunk: Vec,
bitwise_writer: Option>,
}
impl DeflateEncoder {
/// Creates a new Zopfli DEFLATE encoder that will operate according to the
/// specified options.
pub fn new(options: Options, btype: BlockType, sink: W) -> Self {
DeflateEncoder {
options,
btype,
have_chunk: false,
chunk_start: 0,
window_and_chunk: Vec::with_capacity(ZOPFLI_WINDOW_SIZE),
bitwise_writer: Some(BitwiseWriter::new(sink)),
}
}
/// Creates a new Zopfli DEFLATE encoder that operates according to the
/// specified options and is wrapped with a buffer to guarantee that
/// data is compressed in large chunks, which is necessary for decent
/// performance and good compression ratio.
#[cfg(feature = "std")]
pub fn new_buffered(options: Options, btype: BlockType, sink: W) -> std::io::BufWriter {
std::io::BufWriter::with_capacity(
crate::util::ZOPFLI_MASTER_BLOCK_SIZE,
Self::new(options, btype, sink),
)
}
/// Encodes any pending chunks of data and writes them to the sink,
/// consuming the encoder and returning the wrapped sink. The sink
/// will have received a complete DEFLATE stream when this method
/// returns.
///
/// The encoder is automatically [`finish`](Self::finish)ed when
/// dropped, but explicitly finishing it with this method allows
/// handling I/O errors.
pub fn finish(mut self) -> Result {
self._finish().map(|sink| sink.unwrap())
}
/// Compresses the chunk stored at `window_and_chunk`. This includes
/// a rolling window of the last `ZOPFLI_WINDOW_SIZE` data bytes, if
/// available.
#[inline]
fn compress_chunk(&mut self, is_last: bool) -> Result<(), Error> {
deflate_part(
&self.options,
self.btype,
is_last,
&self.window_and_chunk,
self.chunk_start,
self.window_and_chunk.len(),
self.bitwise_writer.as_mut().unwrap(),
)
}
/// Sets the next chunk that will be compressed by the next
/// call to `compress_chunk` and updates the rolling data window
/// accordingly.
fn set_chunk(&mut self, chunk: &[u8]) {
// Remove bytes exceeding the window size. Start with the
// oldest bytes, which are at the beginning of the buffer.
// The buffer length is then the position where the chunk
// we've just received starts
self.window_and_chunk.drain(
..self
.window_and_chunk
.len()
.saturating_sub(ZOPFLI_WINDOW_SIZE),
);
self.chunk_start = self.window_and_chunk.len();
self.window_and_chunk.extend_from_slice(chunk);
self.have_chunk = true;
}
/// Encodes the last chunk and finishes any partial bits.
/// The encoder will be unusable for further compression
/// after this method returns. This is intended to be an
/// implementation detail of the `Drop` trait and
/// [`finish`](Self::finish) method.
fn _finish(&mut self) -> Result, Error> {
if self.bitwise_writer.is_none() {
return Ok(None);
}
self.compress_chunk(true)?;
let mut bitwise_writer = self.bitwise_writer.take().unwrap();
bitwise_writer.finish_partial_bits()?;
Ok(Some(bitwise_writer.out))
}
}
impl Write for DeflateEncoder {
fn write(&mut self, buf: &[u8]) -> Result {
// Any previous chunk is known to be non-last at this point,
// so compress it now
if self.have_chunk {
self.compress_chunk(false)?;
}
// Set the chunk to be used for the next compression operation
// to this chunk. We don't know whether it's last or not yet
self.set_chunk(buf);
Ok(buf.len())
}
fn flush(&mut self) -> Result<(), Error> {
self.bitwise_writer.as_mut().unwrap().out.flush()
}
}
impl Drop for DeflateEncoder {
fn drop(&mut self) {
self._finish().ok();
}
}
// Boilerplate to make latest Rustdoc happy: https://github.com/rust-lang/rust/issues/117796
#[cfg(all(doc, feature = "std"))]
impl std::io::Write for DeflateEncoder {
fn write(&mut self, _buf: &[u8]) -> std::io::Result {
unimplemented!()
}
fn flush(&mut self) -> std::io::Result<()> {
unimplemented!()
}
}
/// Deflate a part, to allow for chunked, streaming compression with [`DeflateEncoder`].
/// It is possible to call this function multiple times in a row, shifting
/// instart and inend to next bytes of the data. If instart is larger than 0, then
/// previous bytes are used as the initial dictionary for LZ77.
/// This function will usually output multiple deflate blocks. If final is true, then
/// the final bit will be set on the last block.
/// Like deflate, but allows to specify start and end byte with instart and
/// inend. Only that part is compressed, but earlier bytes are still used for the
/// back window.
fn deflate_part(
options: &Options,
btype: BlockType,
final_block: bool,
in_data: &[u8],
instart: usize,
inend: usize,
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
/* If btype=Dynamic is specified, it tries all block types. If a lesser btype is
given, then however it forces that one. Neither of the lesser types needs
block splitting as they have no dynamic huffman trees. */
match btype {
BlockType::Uncompressed => {
add_non_compressed_block(final_block, in_data, instart, inend, bitwise_writer)
}
BlockType::Fixed => {
let mut store = Lz77Store::new();
lz77_optimal_fixed(
&mut ZopfliLongestMatchCache::new(inend - instart),
in_data,
instart,
inend,
&mut store,
);
add_lz77_block(
btype,
final_block,
in_data,
&store,
0,
store.size(),
0,
bitwise_writer,
)
}
BlockType::Dynamic => blocksplit_attempt(
options,
final_block,
in_data,
instart,
inend,
bitwise_writer,
),
}
}
/// The type of data blocks to generate for a DEFLATE stream.
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
#[cfg_attr(all(test, feature = "std"), derive(proptest_derive::Arbitrary))]
pub enum BlockType {
/// Non-compressed blocks (BTYPE=00).
///
/// The input data will be divided into chunks up to 64 KiB big and
/// stored in the DEFLATE stream without compression. This is mainly
/// useful for test and development purposes.
Uncompressed,
/// Compressed blocks with fixed Huffman codes (BTYPE=01).
///
/// The input data will be compressed into DEFLATE blocks using a fixed
/// Huffman tree defined in the DEFLATE specification. This provides fast
/// but poor compression, as the Zopfli algorithm is not actually used.
Fixed,
/// Select the most space-efficient block types for the input data.
/// This is the recommended type for the vast majority of Zopfli
/// applications.
///
/// This mode lets the Zopfli algorithm choose the combination of block
/// types that minimizes data size. The emitted block types may be
/// [`Uncompressed`](Self::Uncompressed) or [`Fixed`](Self::Fixed), in
/// addition to compressed with dynamic Huffman codes (BTYPE=10).
Dynamic,
}
impl Default for BlockType {
fn default() -> Self {
Self::Dynamic
}
}
fn fixed_tree() -> (Vec, Vec) {
let mut ll = Vec::with_capacity(ZOPFLI_NUM_LL);
ll.resize(144, 8);
ll.resize(256, 9);
ll.resize(280, 7);
ll.resize(288, 8);
let d = vec![5; ZOPFLI_NUM_D];
(ll, d)
}
/// Changes the population counts in a way that the consequent Huffman tree
/// compression, especially its rle-part, will be more likely to compress this data
/// more efficiently. length contains the size of the histogram.
fn optimize_huffman_for_rle(counts: &mut [usize]) {
let mut length = counts.len();
// 1) We don't want to touch the trailing zeros. We may break the
// rules of the format by adding more data in the distance codes.
loop {
if length == 0 {
return;
}
if counts[length - 1] != 0 {
// Now counts[0..length - 1] does not have trailing zeros.
break;
}
length -= 1;
}
// 2) Let's mark all population counts that already can be encoded
// with an rle code.
let mut good_for_rle = vec![false; length];
// Let's not spoil any of the existing good rle codes.
// Mark any seq of 0's that is longer than 5 as a good_for_rle.
// Mark any seq of non-0's that is longer than 7 as a good_for_rle.
let mut symbol = counts[0];
let mut stride = 0;
for (i, &count) in counts.iter().enumerate().take(length) {
if count != symbol {
if (symbol == 0 && stride >= 5) || (symbol != 0 && stride >= 7) {
for k in 0..stride {
good_for_rle[i - k - 1] = true;
}
}
stride = 1;
symbol = count;
} else {
stride += 1;
}
}
// 3) Let's replace those population counts that lead to more rle codes.
stride = 0;
let mut limit = counts[0];
let mut sum = 0;
for i in 0..(length + 1) {
// Heuristic for selecting the stride ranges to collapse.
if i == length || good_for_rle[i] || (counts[i] as i32 - limit as i32).abs() >= 4 {
if stride >= 4 || (stride >= 3 && sum == 0) {
// The stride must end, collapse what we have, if we have enough (4).
let count = if sum == 0 {
// Don't upgrade an all zeros stride to ones.
0
} else {
cmp::max((sum + stride / 2) / stride, 1)
};
set_counts_to_count(counts, count, i, stride);
}
stride = 0;
sum = 0;
if length > 2 && i < length - 3 {
// All interesting strides have a count of at least 4,
// at least when non-zeros.
limit = (counts[i] + counts[i + 1] + counts[i + 2] + counts[i + 3] + 2) / 4;
} else if i < length {
limit = counts[i];
} else {
limit = 0;
}
}
stride += 1;
if i != length {
sum += counts[i];
}
}
}
// Ensures there are at least 2 distance codes to support buggy decoders.
// Zlib 1.2.1 and below have a bug where it fails if there isn't at least 1
// distance code (with length > 0), even though it's valid according to the
// deflate spec to have 0 distance codes. On top of that, some mobile phones
// require at least two distance codes. To support these decoders too (but
// potentially at the cost of a few bytes), add dummy code lengths of 1.
// References to this bug can be found in the changelog of
// Zlib 1.2.2 and here: http://www.jonof.id.au/forum/index.php?topic=515.0.
//
// d_lengths: the 32 lengths of the distance codes.
fn patch_distance_codes_for_buggy_decoders(d_lengths: &mut [u32]) {
// Ignore the two unused codes from the spec
let num_dist_codes = d_lengths
.iter()
.take(30)
.filter(|&&d_length| d_length != 0)
.count();
match num_dist_codes {
0 => {
d_lengths[0] = 1;
d_lengths[1] = 1;
}
1 => {
let index = if d_lengths[0] == 0 { 0 } else { 1 };
d_lengths[index] = 1;
}
_ => {} // Two or more codes is fine.
}
}
/// Same as `calculate_block_symbol_size`, but for block size smaller than histogram
/// size.
fn calculate_block_symbol_size_small(
ll_lengths: &[u32],
d_lengths: &[u32],
lz77: &Lz77Store,
lstart: usize,
lend: usize,
) -> usize {
let mut result = 0;
debug_assert!(lend == lstart || lend - 1 < lz77.size());
for &item in &lz77.litlens[lstart..lend] {
match item {
LitLen::Literal(litlens_i) => {
debug_assert!(litlens_i < 259);
result += ll_lengths[litlens_i as usize]
}
LitLen::LengthDist(litlens_i, dists_i) => {
debug_assert!(litlens_i < 259);
let ll_symbol = get_length_symbol(litlens_i as usize);
let d_symbol = get_dist_symbol(dists_i);
result += ll_lengths[ll_symbol];
result += d_lengths[d_symbol];
result += get_length_symbol_extra_bits(ll_symbol);
result += get_dist_symbol_extra_bits(d_symbol);
}
}
}
result += ll_lengths[256]; // end symbol
result as usize
}
/// Same as `calculate_block_symbol_size`, but with the histogram provided by the caller.
fn calculate_block_symbol_size_given_counts(
ll_counts: &[usize],
d_counts: &[usize],
ll_lengths: &[u32],
d_lengths: &[u32],
lz77: &Lz77Store,
lstart: usize,
lend: usize,
) -> usize {
if lstart + ZOPFLI_NUM_LL * 3 > lend {
calculate_block_symbol_size_small(ll_lengths, d_lengths, lz77, lstart, lend)
} else {
let mut result = 0;
for i in 0..256 {
result += ll_lengths[i] * ll_counts[i] as u32;
}
for i in 257..286 {
result += ll_lengths[i] * ll_counts[i] as u32;
result += (get_length_symbol_extra_bits(i) as usize * ll_counts[i]) as u32;
}
for i in 0..30 {
result += d_lengths[i] * d_counts[i] as u32;
result += (get_dist_symbol_extra_bits(i) as usize * d_counts[i]) as u32;
}
result += ll_lengths[256]; // end symbol
result as usize
}
}
/// Calculates size of the part after the header and tree of an LZ77 block, in bits.
fn calculate_block_symbol_size(
ll_lengths: &[u32],
d_lengths: &[u32],
lz77: &Lz77Store,
lstart: usize,
lend: usize,
) -> usize {
if lstart + ZOPFLI_NUM_LL * 3 > lend {
calculate_block_symbol_size_small(ll_lengths, d_lengths, lz77, lstart, lend)
} else {
let (ll_counts, d_counts) = lz77.get_histogram(lstart, lend);
calculate_block_symbol_size_given_counts(
&*ll_counts,
&*d_counts,
ll_lengths,
d_lengths,
lz77,
lstart,
lend,
)
}
}
/// Encodes the Huffman tree and returns how many bits its encoding takes; only returns the size
/// and runs faster.
fn encode_tree_no_output(
ll_lengths: &[u32],
d_lengths: &[u32],
use_16: bool,
use_17: bool,
use_18: bool,
) -> usize {
let mut hlit = 29; /* 286 - 257 */
let mut hdist = 29; /* 32 - 1, but gzip does not like hdist > 29.*/
let mut clcounts = [0; 19];
/* The order in which code length code lengths are encoded as per deflate. */
let order = [
16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15,
];
let mut result_size = 0;
/* Trim zeros. */
while hlit > 0 && ll_lengths[257 + hlit - 1] == 0 {
hlit -= 1;
}
while hdist > 0 && d_lengths[1 + hdist - 1] == 0 {
hdist -= 1;
}
let hlit2 = hlit + 257;
let lld_total = hlit2 + hdist + 1; /* Total amount of literal, length, distance codes. */
let mut i = 0;
while i < lld_total {
/* This is an encoding of a huffman tree, so now the length is a symbol */
let symbol = if i < hlit2 {
ll_lengths[i]
} else {
d_lengths[i - hlit2]
} as u8;
let mut count = 1;
if use_16 || (symbol == 0 && (use_17 || use_18)) {
let mut j = i + 1;
let mut symbol_calc = if j < hlit2 {
ll_lengths[j]
} else {
d_lengths[j - hlit2]
} as u8;
while j < lld_total && symbol == symbol_calc {
count += 1;
j += 1;
symbol_calc = if j < hlit2 {
ll_lengths[j]
} else {
d_lengths[j - hlit2]
} as u8;
}
}
i += count - 1;
/* Repetitions of zeroes */
if symbol == 0 && count >= 3 {
if use_18 {
while count >= 11 {
let count2 = if count > 138 { 138 } else { count };
clcounts[18] += 1;
count -= count2;
}
}
if use_17 {
while count >= 3 {
let count2 = if count > 10 { 10 } else { count };
clcounts[17] += 1;
count -= count2;
}
}
}
/* Repetitions of any symbol */
if use_16 && count >= 4 {
count -= 1; /* Since the first one is hardcoded. */
clcounts[symbol as usize] += 1;
while count >= 3 {
let count2 = if count > 6 { 6 } else { count };
clcounts[16] += 1;
count -= count2;
}
}
/* No or insufficient repetition */
clcounts[symbol as usize] += count;
while count > 0 {
count -= 1;
}
i += 1;
}
let clcl = length_limited_code_lengths(&clcounts, 7);
let mut hclen = 15;
/* Trim zeros. */
while hclen > 0 && clcounts[order[hclen + 4 - 1]] == 0 {
hclen -= 1;
}
result_size += 14; /* hlit, hdist, hclen bits */
result_size += (hclen + 4) * 3; /* clcl bits */
for i in 0..19 {
result_size += clcl[i] as usize * clcounts[i];
}
/* Extra bits. */
result_size += clcounts[16] * 2;
result_size += clcounts[17] * 3;
result_size += clcounts[18] * 7;
result_size
}
static TRUTH_TABLE: [(bool, bool, bool); 8] = [
(false, false, false),
(true, false, false),
(false, true, false),
(true, true, false),
(false, false, true),
(true, false, true),
(false, true, true),
(true, true, true),
];
/// Gives the exact size of the tree, in bits, as it will be encoded in DEFLATE.
fn calculate_tree_size(ll_lengths: &[u32], d_lengths: &[u32]) -> usize {
TRUTH_TABLE
.iter()
.map(|&(use_16, use_17, use_18)| {
encode_tree_no_output(ll_lengths, d_lengths, use_16, use_17, use_18)
})
.min()
.unwrap_or(0)
}
/// Encodes the Huffman tree and returns how many bits its encoding takes and returns output.
// TODO: This return value is unused.
fn encode_tree(
ll_lengths: &[u32],
d_lengths: &[u32],
use_16: bool,
use_17: bool,
use_18: bool,
bitwise_writer: &mut BitwiseWriter,
) -> Result {
let mut hlit = 29; /* 286 - 257 */
let mut hdist = 29; /* 32 - 1, but gzip does not like hdist > 29.*/
let mut clcounts = [0; 19];
/* The order in which code length code lengths are encoded as per deflate. */
let order = [
16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15,
];
let mut result_size = 0;
let mut rle = vec![];
let mut rle_bits = vec![];
/* Trim zeros. */
while hlit > 0 && ll_lengths[257 + hlit - 1] == 0 {
hlit -= 1;
}
while hdist > 0 && d_lengths[1 + hdist - 1] == 0 {
hdist -= 1;
}
let hlit2 = hlit + 257;
let lld_total = hlit2 + hdist + 1; /* Total amount of literal, length, distance codes. */
let mut i = 0;
while i < lld_total {
/* This is an encoding of a huffman tree, so now the length is a symbol */
let symbol = if i < hlit2 {
ll_lengths[i]
} else {
d_lengths[i - hlit2]
} as u8;
let mut count = 1;
if use_16 || (symbol == 0 && (use_17 || use_18)) {
let mut j = i + 1;
let mut symbol_calc = if j < hlit2 {
ll_lengths[j]
} else {
d_lengths[j - hlit2]
} as u8;
while j < lld_total && symbol == symbol_calc {
count += 1;
j += 1;
symbol_calc = if j < hlit2 {
ll_lengths[j]
} else {
d_lengths[j - hlit2]
} as u8;
}
}
i += count - 1;
/* Repetitions of zeroes */
if symbol == 0 && count >= 3 {
if use_18 {
while count >= 11 {
let count2 = if count > 138 { 138 } else { count };
rle.push(18);
rle_bits.push(count2 - 11);
clcounts[18] += 1;
count -= count2;
}
}
if use_17 {
while count >= 3 {
let count2 = if count > 10 { 10 } else { count };
rle.push(17);
rle_bits.push(count2 - 3);
clcounts[17] += 1;
count -= count2;
}
}
}
/* Repetitions of any symbol */
if use_16 && count >= 4 {
count -= 1; /* Since the first one is hardcoded. */
clcounts[symbol as usize] += 1;
rle.push(symbol);
rle_bits.push(0);
while count >= 3 {
let count2 = if count > 6 { 6 } else { count };
rle.push(16);
rle_bits.push(count2 - 3);
clcounts[16] += 1;
count -= count2;
}
}
/* No or insufficient repetition */
clcounts[symbol as usize] += count;
while count > 0 {
rle.push(symbol);
rle_bits.push(0);
count -= 1;
}
i += 1;
}
let clcl = length_limited_code_lengths(&clcounts, 7);
let clsymbols = lengths_to_symbols(&clcl, 7);
let mut hclen = 15;
/* Trim zeros. */
while hclen > 0 && clcounts[order[hclen + 4 - 1]] == 0 {
hclen -= 1;
}
bitwise_writer.add_bits(hlit as u32, 5)?;
bitwise_writer.add_bits(hdist as u32, 5)?;
bitwise_writer.add_bits(hclen as u32, 4)?;
for &item in order.iter().take(hclen + 4) {
bitwise_writer.add_bits(clcl[item], 3)?;
}
for i in 0..rle.len() {
let rle_i = rle[i] as usize;
let rle_bits_i = rle_bits[i] as u32;
let sym = clsymbols[rle_i];
bitwise_writer.add_huffman_bits(sym, clcl[rle_i])?;
/* Extra bits. */
if rle_i == 16 {
bitwise_writer.add_bits(rle_bits_i, 2)?;
} else if rle_i == 17 {
bitwise_writer.add_bits(rle_bits_i, 3)?;
} else if rle_i == 18 {
bitwise_writer.add_bits(rle_bits_i, 7)?;
}
}
result_size += 14; /* hlit, hdist, hclen bits */
result_size += (hclen + 4) * 3; /* clcl bits */
for i in 0..19 {
result_size += clcl[i] as usize * clcounts[i];
}
/* Extra bits. */
result_size += clcounts[16] * 2;
result_size += clcounts[17] * 3;
result_size += clcounts[18] * 7;
Ok(result_size)
}
fn add_dynamic_tree(
ll_lengths: &[u32],
d_lengths: &[u32],
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
let mut best = 0;
let mut bestsize = 0;
for i in 0..8 {
let size = encode_tree_no_output(ll_lengths, d_lengths, i & 1 > 0, i & 2 > 0, i & 4 > 0);
if bestsize == 0 || size < bestsize {
bestsize = size;
best = i;
}
}
encode_tree(
ll_lengths,
d_lengths,
best & 1 > 0,
best & 2 > 0,
best & 4 > 0,
bitwise_writer,
)
.map(|_| ())
}
/// Adds a deflate block with the given LZ77 data to the output.
/// `options`: global program options
/// `btype`: the block type, must be `Fixed` or `Dynamic`
/// `final`: whether to set the "final" bit on this block, must be the last block
/// `litlens`: literal/length array of the LZ77 data, in the same format as in
/// `Lz77Store`.
/// `dists`: distance array of the LZ77 data, in the same format as in
/// `Lz77Store`.
/// `lstart`: where to start in the LZ77 data
/// `lend`: where to end in the LZ77 data (not inclusive)
/// `expected_data_size`: the uncompressed block size, used for assert, but you can
/// set it to `0` to not do the assertion.
/// `bitwise_writer`: writer responsible for appending bits
#[allow(clippy::too_many_arguments)] // Not feasible to refactor in a more readable way
fn add_lz77_block(
btype: BlockType,
final_block: bool,
in_data: &[u8],
lz77: &Lz77Store,
lstart: usize,
lend: usize,
expected_data_size: usize,
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
if btype == BlockType::Uncompressed {
let length = lz77.get_byte_range(lstart, lend);
let pos = if lstart == lend { 0 } else { lz77.pos[lstart] };
let end = pos + length;
return add_non_compressed_block(final_block, in_data, pos, end, bitwise_writer);
}
bitwise_writer.add_bit(final_block as u8)?;
let (ll_lengths, d_lengths) = match btype {
BlockType::Uncompressed => unreachable!(),
BlockType::Fixed => {
bitwise_writer.add_bit(1)?;
bitwise_writer.add_bit(0)?;
fixed_tree()
}
BlockType::Dynamic => {
bitwise_writer.add_bit(0)?;
bitwise_writer.add_bit(1)?;
let (_, ll_lengths, d_lengths) = get_dynamic_lengths(lz77, lstart, lend);
let _detect_tree_size = bitwise_writer.bytes_written();
add_dynamic_tree(&ll_lengths, &d_lengths, bitwise_writer)?;
debug!(
"treesize: {}",
bitwise_writer.bytes_written() - _detect_tree_size
);
(ll_lengths, d_lengths)
}
};
let ll_symbols = lengths_to_symbols(&ll_lengths, 15);
let d_symbols = lengths_to_symbols(&d_lengths, 15);
let detect_block_size = bitwise_writer.bytes_written();
add_lz77_data(
lz77,
lstart,
lend,
expected_data_size,
&ll_symbols,
&ll_lengths,
&d_symbols,
&d_lengths,
bitwise_writer,
)?;
/* End symbol. */
bitwise_writer.add_huffman_bits(ll_symbols[256], ll_lengths[256])?;
if log_enabled!(log::Level::Debug) {
let _uncompressed_size = lz77.litlens[lstart..lend]
.iter()
.fold(0, |acc, &x| acc + x.size());
let _compressed_size = bitwise_writer.bytes_written() - detect_block_size;
debug!(
"compressed block size: {} ({}k) (unc: {})",
_compressed_size,
_compressed_size / 1024,
_uncompressed_size
);
}
Ok(())
}
/// Calculates block size in bits.
/// litlens: lz77 lit/lengths
/// dists: ll77 distances
/// lstart: start of block
/// lend: end of block (not inclusive)
pub fn calculate_block_size(lz77: &Lz77Store, lstart: usize, lend: usize, btype: BlockType) -> f64 {
match btype {
BlockType::Uncompressed => {
let length = lz77.get_byte_range(lstart, lend);
let rem = length % 65535;
let blocks = length / 65535 + (if rem > 0 { 1 } else { 0 });
/* An uncompressed block must actually be split into multiple blocks if it's
larger than 65535 bytes long. Eeach block header is 5 bytes: 3 bits,
padding, LEN and NLEN (potential less padding for first one ignored). */
(blocks * 5 * 8 + length * 8) as f64
}
BlockType::Fixed => {
let fixed_tree = fixed_tree();
let ll_lengths = fixed_tree.0;
let d_lengths = fixed_tree.1;
let mut result = 3.0; /* bfinal and btype bits */
result +=
calculate_block_symbol_size(&ll_lengths, &d_lengths, lz77, lstart, lend) as f64;
result
}
BlockType::Dynamic => get_dynamic_lengths(lz77, lstart, lend).0 + 3.0,
}
}
/// Tries out `OptimizeHuffmanForRle` for this block, if the result is smaller,
/// uses it, otherwise keeps the original. Returns size of encoded tree and data in
/// bits, not including the 3-bit block header.
fn try_optimize_huffman_for_rle(
lz77: &Lz77Store,
lstart: usize,
lend: usize,
ll_counts: &[usize],
d_counts: &[usize],
ll_lengths: Vec,
d_lengths: Vec,
) -> (f64, Vec, Vec) {
let mut ll_counts2 = Vec::from(ll_counts);
let mut d_counts2 = Vec::from(d_counts);
let treesize = calculate_tree_size(&ll_lengths, &d_lengths);
let datasize = calculate_block_symbol_size_given_counts(
ll_counts,
d_counts,
&ll_lengths,
&d_lengths,
lz77,
lstart,
lend,
);
optimize_huffman_for_rle(&mut ll_counts2);
optimize_huffman_for_rle(&mut d_counts2);
let ll_lengths2 = length_limited_code_lengths(&ll_counts2, 15);
let mut d_lengths2 = length_limited_code_lengths(&d_counts2, 15);
patch_distance_codes_for_buggy_decoders(&mut d_lengths2[..]);
let treesize2 = calculate_tree_size(&ll_lengths2, &d_lengths2);
let datasize2 = calculate_block_symbol_size_given_counts(
ll_counts,
d_counts,
&ll_lengths2,
&d_lengths2,
lz77,
lstart,
lend,
);
if treesize2 + datasize2 < treesize + datasize {
(((treesize2 + datasize2) as f64), ll_lengths2, d_lengths2)
} else {
((treesize + datasize) as f64, ll_lengths, d_lengths)
}
}
/// Calculates the bit lengths for the symbols for dynamic blocks. Chooses bit
/// lengths that give the smallest size of tree encoding + encoding of all the
/// symbols to have smallest output size. This are not necessarily the ideal Huffman
/// bit lengths. Returns size of encoded tree and data in bits, not including the
/// 3-bit block header.
fn get_dynamic_lengths(lz77: &Lz77Store, lstart: usize, lend: usize) -> (f64, Vec, Vec) {
let (mut ll_counts, d_counts) = lz77.get_histogram(lstart, lend);
ll_counts[256] = 1; /* End symbol. */
let ll_lengths = length_limited_code_lengths(&*ll_counts, 15);
let mut d_lengths = length_limited_code_lengths(&*d_counts, 15);
patch_distance_codes_for_buggy_decoders(&mut d_lengths[..]);
try_optimize_huffman_for_rle(
lz77,
lstart,
lend,
&*ll_counts,
&*d_counts,
ll_lengths,
d_lengths,
)
}
/// Adds all lit/len and dist codes from the lists as huffman symbols. Does not add
/// end code 256. `expected_data_size` is the uncompressed block size, used for
/// assert, but you can set it to `0` to not do the assertion.
#[allow(clippy::too_many_arguments)] // Not feasible to refactor in a more readable way
fn add_lz77_data(
lz77: &Lz77Store,
lstart: usize,
lend: usize,
expected_data_size: usize,
ll_symbols: &[u32],
ll_lengths: &[u32],
d_symbols: &[u32],
d_lengths: &[u32],
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
let mut testlength = 0;
for &item in &lz77.litlens[lstart..lend] {
match item {
LitLen::Literal(lit) => {
let litlen = lit as usize;
debug_assert!(litlen < 256);
debug_assert!(ll_lengths[litlen] > 0);
bitwise_writer.add_huffman_bits(ll_symbols[litlen], ll_lengths[litlen])?;
testlength += 1;
}
LitLen::LengthDist(len, dist) => {
let litlen = len as usize;
let lls = get_length_symbol(litlen);
let ds = get_dist_symbol(dist);
debug_assert!((3..=288).contains(&litlen));
debug_assert!(ll_lengths[lls] > 0);
debug_assert!(d_lengths[ds] > 0);
bitwise_writer.add_huffman_bits(ll_symbols[lls], ll_lengths[lls])?;
bitwise_writer.add_bits(
get_length_extra_bits_value(litlen),
get_length_extra_bits(litlen) as u32,
)?;
bitwise_writer.add_huffman_bits(d_symbols[ds], d_lengths[ds])?;
bitwise_writer.add_bits(
get_dist_extra_bits_value(dist) as u32,
get_dist_extra_bits(dist) as u32,
)?;
testlength += litlen;
}
}
}
debug_assert!(expected_data_size == 0 || testlength == expected_data_size);
Ok(())
}
#[allow(clippy::too_many_arguments)] // Not feasible to refactor in a more readable way
fn add_lz77_block_auto_type(
final_block: bool,
in_data: &[u8],
lz77: &Lz77Store,
lstart: usize,
lend: usize,
expected_data_size: usize,
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
let uncompressedcost = calculate_block_size(lz77, lstart, lend, BlockType::Uncompressed);
let mut fixedcost = calculate_block_size(lz77, lstart, lend, BlockType::Fixed);
let dyncost = calculate_block_size(lz77, lstart, lend, BlockType::Dynamic);
/* Whether to perform the expensive calculation of creating an optimal block
with fixed huffman tree to check if smaller. Only do this for small blocks or
blocks which already are pretty good with fixed huffman tree. */
let expensivefixed = (lz77.size() < 1000) || fixedcost <= dyncost * 1.1;
let mut fixedstore = Lz77Store::new();
if lstart == lend {
/* Smallest empty block is represented by fixed block */
bitwise_writer.add_bits(final_block as u32, 1)?;
bitwise_writer.add_bits(1, 2)?; /* btype 01 */
bitwise_writer.add_bits(0, 7)?; /* end symbol has code 0000000 */
return Ok(());
}
if expensivefixed {
/* Recalculate the LZ77 with lz77_optimal_fixed */
let instart = lz77.pos[lstart];
let inend = instart + lz77.get_byte_range(lstart, lend);
lz77_optimal_fixed(
&mut ZopfliLongestMatchCache::new(inend - instart),
in_data,
instart,
inend,
&mut fixedstore,
);
fixedcost = calculate_block_size(&fixedstore, 0, fixedstore.size(), BlockType::Fixed);
}
if uncompressedcost <= fixedcost && uncompressedcost <= dyncost {
add_lz77_block(
BlockType::Uncompressed,
final_block,
in_data,
lz77,
lstart,
lend,
expected_data_size,
bitwise_writer,
)
} else if fixedcost <= dyncost {
if expensivefixed {
add_lz77_block(
BlockType::Fixed,
final_block,
in_data,
&fixedstore,
0,
fixedstore.size(),
expected_data_size,
bitwise_writer,
)
} else {
add_lz77_block(
BlockType::Fixed,
final_block,
in_data,
lz77,
lstart,
lend,
expected_data_size,
bitwise_writer,
)
}
} else {
add_lz77_block(
BlockType::Dynamic,
final_block,
in_data,
lz77,
lstart,
lend,
expected_data_size,
bitwise_writer,
)
}
}
/// Calculates block size in bits, automatically using the best btype.
pub fn calculate_block_size_auto_type(lz77: &Lz77Store, lstart: usize, lend: usize) -> f64 {
let uncompressedcost = calculate_block_size(lz77, lstart, lend, BlockType::Uncompressed);
/* Don't do the expensive fixed cost calculation for larger blocks that are
unlikely to use it. */
let fixedcost = if lz77.size() > 1000 {
uncompressedcost
} else {
calculate_block_size(lz77, lstart, lend, BlockType::Fixed)
};
let dyncost = calculate_block_size(lz77, lstart, lend, BlockType::Dynamic);
uncompressedcost.min(fixedcost).min(dyncost)
}
fn add_all_blocks(
splitpoints: &[usize],
lz77: &Lz77Store,
final_block: bool,
in_data: &[u8],
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
let mut last = 0;
for &item in splitpoints.iter() {
add_lz77_block_auto_type(false, in_data, lz77, last, item, 0, bitwise_writer)?;
last = item;
}
add_lz77_block_auto_type(
final_block,
in_data,
lz77,
last,
lz77.size(),
0,
bitwise_writer,
)
}
fn blocksplit_attempt(
options: &Options,
final_block: bool,
in_data: &[u8],
instart: usize,
inend: usize,
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
let mut totalcost = 0.0;
let mut lz77 = Lz77Store::new();
/* byte coordinates rather than lz77 index */
let mut splitpoints_uncompressed = Vec::with_capacity(options.maximum_block_splits as usize);
blocksplit(
in_data,
instart,
inend,
options.maximum_block_splits,
&mut splitpoints_uncompressed,
);
let npoints = splitpoints_uncompressed.len();
let mut splitpoints = Vec::with_capacity(npoints);
let mut last = instart;
for &item in &splitpoints_uncompressed {
let store = lz77_optimal(
&mut ZopfliLongestMatchCache::new(item - last),
in_data,
last,
item,
options.iteration_count.get(),
options.iterations_without_improvement.get(),
);
totalcost += calculate_block_size_auto_type(&store, 0, store.size());
// ZopfliAppendLZ77Store(&store, &lz77);
debug_assert!(instart == inend || store.size() > 0);
for (&litlens, &pos) in store.litlens.iter().zip(store.pos.iter()) {
lz77.append_store_item(litlens, pos);
}
splitpoints.push(lz77.size());
last = item;
}
let store = lz77_optimal(
&mut ZopfliLongestMatchCache::new(inend - last),
in_data,
last,
inend,
options.iteration_count.get(),
options.iterations_without_improvement.get(),
);
totalcost += calculate_block_size_auto_type(&store, 0, store.size());
// ZopfliAppendLZ77Store(&store, &lz77);
debug_assert!(instart == inend || store.size() > 0);
for (&litlens, &pos) in store.litlens.iter().zip(store.pos.iter()) {
lz77.append_store_item(litlens, pos);
}
/* Second block splitting attempt */
if npoints > 1 {
let mut splitpoints2 = Vec::with_capacity(splitpoints_uncompressed.len());
let mut totalcost2 = 0.0;
blocksplit_lz77(&lz77, options.maximum_block_splits, &mut splitpoints2);
let mut last = 0;
for &item in &splitpoints2 {
totalcost2 += calculate_block_size_auto_type(&lz77, last, item);
last = item;
}
totalcost2 += calculate_block_size_auto_type(&lz77, last, lz77.size());
if totalcost2 < totalcost {
splitpoints = splitpoints2;
}
}
add_all_blocks(&splitpoints, &lz77, final_block, in_data, bitwise_writer)
}
/// Since an uncompressed block can be max 65535 in size, it actually adds
/// multiple blocks if needed.
fn add_non_compressed_block(
final_block: bool,
in_data: &[u8],
instart: usize,
inend: usize,
bitwise_writer: &mut BitwiseWriter,
) -> Result<(), Error> {
let in_data = &in_data[instart..inend];
let in_data_chunks = in_data.chunks(65535).size_hint().0;
for (chunk, is_final) in in_data
.chunks(65535)
.flag_last()
// Make sure that we output at least one chunk if this is the final block
.chain(iter::once((&[][..], true)))
.take(if final_block {
cmp::max(in_data_chunks, 1)
} else {
in_data_chunks
})
{
let blocksize = chunk.len();
let nlen = !blocksize;
bitwise_writer.add_bit((final_block && is_final) as u8)?;
/* BTYPE 00 */
bitwise_writer.add_bit(0)?;
bitwise_writer.add_bit(0)?;
bitwise_writer.finish_partial_bits()?;
bitwise_writer.add_byte((blocksize % 256) as u8)?;
bitwise_writer.add_byte(((blocksize / 256) % 256) as u8)?;
bitwise_writer.add_byte((nlen % 256) as u8)?;
bitwise_writer.add_byte(((nlen / 256) % 256) as u8)?;
bitwise_writer.add_bytes(chunk)?;
}
Ok(())
}
struct BitwiseWriter {
bit: u8,
bp: u8,
len: usize,
out: W,
}
impl BitwiseWriter {
fn new(out: W) -> BitwiseWriter {
BitwiseWriter {
bit: 0,
bp: 0,
len: 0,
out,
}
}
fn bytes_written(&self) -> usize {
self.len + if self.bp > 0 { 1 } else { 0 }
}
/// For when you want to add a full byte.
fn add_byte(&mut self, byte: u8) -> Result<(), Error> {
self.add_bytes(&[byte])
}
/// For adding a slice of bytes.
fn add_bytes(&mut self, bytes: &[u8]) -> Result<(), Error> {
self.len += bytes.len();
self.out.write_all(bytes)
}
fn add_bit(&mut self, bit: u8) -> Result<(), Error> {
self.bit |= bit << self.bp;
self.bp += 1;
if self.bp == 8 {
self.finish_partial_bits()
} else {
Ok(())
}
}
fn add_bits(&mut self, symbol: u32, length: u32) -> Result<(), Error> {
// TODO: make more efficient (add more bits at once)
for i in 0..length {
let bit = ((symbol >> i) & 1) as u8;
self.add_bit(bit)?;
}
Ok(())
}
/// Adds bits, like `add_bits`, but the order is inverted. The deflate specification
/// uses both orders in one standard.
fn add_huffman_bits(&mut self, symbol: u32, length: u32) -> Result<(), Error> {
// TODO: make more efficient (add more bits at once)
for i in 0..length {
let bit = ((symbol >> (length - i - 1)) & 1) as u8;
self.add_bit(bit)?;
}
Ok(())
}
fn finish_partial_bits(&mut self) -> Result<(), Error> {
if self.bp != 0 {
let bytes = &[self.bit];
self.add_bytes(bytes)?;
self.bit = 0;
self.bp = 0;
}
Ok(())
}
}
fn set_counts_to_count(counts: &mut [usize], count: usize, i: usize, stride: usize) {
for c in &mut counts[(i - stride)..i] {
*c = count;
}
}
#[cfg(test)]
mod test {
use miniz_oxide::inflate;
use super::*;
#[test]
fn test_set_counts_to_count() {
let mut counts = vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 9];
let count = 100;
let i = 8;
let stride = 5;
set_counts_to_count(&mut counts, count, i, stride);
assert_eq!(counts, vec![0, 1, 2, 100, 100, 100, 100, 100, 8, 9])
}
#[test]
fn weird_encoder_write_size_combinations_works() {
let mut compressed_data = vec![];
let default_options = Options::default();
let mut encoder =
DeflateEncoder::new(default_options, BlockType::default(), &mut compressed_data);
encoder.write_all(&[0]).unwrap();
encoder.write_all(&[]).unwrap();
encoder.write_all(&[1, 2]).unwrap();
encoder.write_all(&[]).unwrap();
encoder.write_all(&[]).unwrap();
encoder.write_all(&[3]).unwrap();
encoder.write_all(&[4]).unwrap();
encoder.finish().unwrap();
let decompressed_data = inflate::decompress_to_vec(&compressed_data)
.expect("Could not inflate compressed stream");
assert_eq!(
&[0, 1, 2, 3, 4][..],
decompressed_data,
"Decompressed data should match input data"
);
}
}
zopfli-0.8.1/src/gzip.rs 0000644 0000000 0000000 00000007451 10461020230 0013252 0 ustar 0000000 0000000 use crate::{BlockType, DeflateEncoder, Error, Options, Write};
/// A Gzip encoder powered by the Zopfli algorithm, that compresses data using
/// a [`DeflateEncoder`]. Most users will find using [`compress`](crate::compress)
/// easier and more performant.
///
/// The caveats about short writes in [`DeflateEncoder`]s carry over to `GzipEncoder`s:
/// for best performance and compression, it is best to avoid them. One way to ensure
/// this is to use the [`new_buffered`](GzipEncoder::new_buffered) method.
pub struct GzipEncoder {
deflate_encoder: Option>,
crc32_hasher: crc32fast::Hasher,
input_size: u32,
}
impl GzipEncoder {
/// Creates a new Gzip encoder that will operate according to the
/// specified options.
pub fn new(options: Options, btype: BlockType, mut sink: W) -> Result {
static HEADER: &[u8] = &[
31, // ID1
139, // ID2
8, // CM
0, // FLG
0, // MTIME
0, 0, 0, 2, // XFL, 2 indicates best compression.
3, // OS follows Unix conventions.
];
sink.write_all(HEADER)?;
Ok(Self {
deflate_encoder: Some(DeflateEncoder::new(options, btype, sink)),
crc32_hasher: crc32fast::Hasher::new(),
input_size: 0,
})
}
/// Creates a new Gzip encoder that operates according to the specified
/// options and is wrapped with a buffer to guarantee that data is
/// compressed in large chunks, which is necessary for decent performance
/// and good compression ratio.
#[cfg(feature = "std")]
pub fn new_buffered(
options: Options,
btype: BlockType,
sink: W,
) -> Result, Error> {
Ok(std::io::BufWriter::with_capacity(
crate::util::ZOPFLI_MASTER_BLOCK_SIZE,
Self::new(options, btype, sink)?,
))
}
/// Encodes any pending chunks of data and writes them to the sink,
/// consuming the encoder and returning the wrapped sink. The sink
/// will have received a complete Gzip stream when this method
/// returns.
///
/// The encoder is automatically [`finish`](Self::finish)ed when
/// dropped, but explicitly finishing it with this method allows
/// handling I/O errors.
pub fn finish(mut self) -> Result {
self._finish().map(|sink| sink.unwrap())
}
fn _finish(&mut self) -> Result, Error> {
if self.deflate_encoder.is_none() {
return Ok(None);
}
let mut sink = self.deflate_encoder.take().unwrap().finish()?;
sink.write_all(&self.crc32_hasher.clone().finalize().to_le_bytes())?;
sink.write_all(&self.input_size.to_le_bytes())?;
Ok(Some(sink))
}
}
impl Write for GzipEncoder {
fn write(&mut self, buf: &[u8]) -> Result {
self.deflate_encoder
.as_mut()
.unwrap()
.write(buf)
.map(|bytes_written| {
self.crc32_hasher.update(&buf[..bytes_written]);
self.input_size = self.input_size.wrapping_add(bytes_written as u32);
bytes_written
})
}
fn flush(&mut self) -> Result<(), Error> {
self.deflate_encoder.as_mut().unwrap().flush()
}
}
impl Drop for GzipEncoder {
fn drop(&mut self) {
self._finish().ok();
}
}
// Boilerplate to make latest Rustdoc happy: https://github.com/rust-lang/rust/issues/117796
#[cfg(all(doc, feature = "std"))]
impl std::io::Write for GzipEncoder {
fn write(&mut self, _buf: &[u8]) -> std::io::Result {
unimplemented!()
}
fn flush(&mut self) -> std::io::Result<()> {
unimplemented!()
}
}
zopfli-0.8.1/src/hash.rs 0000644 0000000 0000000 00000015255 10461020230 0013225 0 ustar 0000000 0000000 use alloc::{
alloc::{alloc, handle_alloc_error, Layout},
boxed::Box,
};
use core::ptr::{addr_of, addr_of_mut, NonNull};
#[cfg(feature = "std")]
use lockfree_object_pool::LinearObjectPool;
#[cfg(feature = "std")]
use once_cell::sync::Lazy;
use crate::util::{ZOPFLI_MIN_MATCH, ZOPFLI_WINDOW_MASK, ZOPFLI_WINDOW_SIZE};
const HASH_SHIFT: i32 = 5;
const HASH_MASK: u16 = 32767;
#[derive(Copy, Clone, PartialEq, Eq)]
pub enum Which {
Hash1,
Hash2,
}
#[derive(Clone)]
pub struct SmallerHashThing {
prev: u16, /* Index to index of prev. occurrence of same hash. */
hashval: Option, /* Index to hash value at this index. */
}
#[derive(Clone)]
pub struct HashThing {
head: [i16; 65536], /* Hash value to index of its most recent occurrence. */
prev_and_hashval: [SmallerHashThing; ZOPFLI_WINDOW_SIZE],
val: u16, /* Current hash value. */
}
impl HashThing {
fn update(&mut self, hpos: usize) {
let hashval = self.val;
let index = self.val as usize;
let head_index = self.head[index];
let prev = if head_index >= 0
&& self.prev_and_hashval[head_index as usize]
.hashval
.map_or(false, |hv| hv == self.val)
{
head_index as u16
} else {
hpos as u16
};
self.prev_and_hashval[hpos] = SmallerHashThing {
prev,
hashval: Some(hashval),
};
self.head[index] = hpos as i16;
}
}
#[derive(Clone)]
pub struct ZopfliHash {
hash1: HashThing,
hash2: HashThing,
pub same: [u16; ZOPFLI_WINDOW_SIZE], /* Amount of repetitions of same byte after this .*/
}
impl ZopfliHash {
pub fn new() -> Box {
const LAYOUT: Layout = Layout::new::();
let ptr = NonNull::new(unsafe { alloc(LAYOUT) } as *mut ZopfliHash)
.unwrap_or_else(|| handle_alloc_error(LAYOUT));
unsafe {
Self::init(ptr);
Box::from_raw(ptr.as_ptr())
}
}
/// Initializes the [`ZopfliHash`] instance pointed by `hash` to an initial state.
///
/// ## Safety
/// `hash` must point to aligned, valid memory for writes.
unsafe fn init(hash: NonNull) {
let hash = hash.as_ptr();
// SAFETY: addr_of(_mut) macros are used to avoid creating intermediate references, which
// are undefined behavior when data is uninitialized. Note that it also is UB to
// assume that integer values and arrays can be read after allocating their memory:
// the allocator returns valid, but uninitialized pointers that are not guaranteed
// to hold a fixed bit pattern (c.f. core::mem::MaybeUnit docs and
// https://doc.rust-lang.org/std/ptr/index.html#safety).
for i in 0..ZOPFLI_WINDOW_SIZE {
// Arrays are guaranteed to be laid out with their elements placed in consecutive
// memory positions: https://doc.rust-lang.org/reference/type-layout.html#array-layout.
// Therefore, a pointer to an array has the same address as the pointer to its first
// element, and adding size_of::() bytes to that address yields the address of the
// second element, and so on.
let prev_and_hashval =
(addr_of_mut!((*hash).hash1.prev_and_hashval) as *mut SmallerHashThing).add(i);
addr_of_mut!((*prev_and_hashval).prev).write(i as u16);
addr_of_mut!((*prev_and_hashval).hashval).write(None);
}
// Rust signed integers are guaranteed to be represented in two's complement notation:
// https://doc.rust-lang.org/reference/types/numeric.html#integer-types
// In this notation, -1 is expressed as an all-ones value. Therefore, writing
// size_of::<[i16; N]> all-ones bytes initializes all of them to -1.
addr_of_mut!((*hash).hash1.head).write_bytes(0xFF, 1);
addr_of_mut!((*hash).hash1.val).write(0);
addr_of_mut!((*hash).hash2).copy_from_nonoverlapping(addr_of!((*hash).hash1), 1);
// Zero-initializes all the array elements
addr_of_mut!((*hash).same).write_bytes(0, 1);
}
pub fn reset(&mut self) {
unsafe { Self::init(NonNull::new(self).unwrap()) }
}
pub fn warmup(&mut self, arr: &[u8], pos: usize, end: usize) {
let c = arr[pos];
self.update_val(c);
if pos + 1 < end {
let c = arr[pos + 1];
self.update_val(c);
}
}
/// Update the sliding hash value with the given byte. All calls to this function
/// must be made on consecutive input characters. Since the hash value exists out
/// of multiple input bytes, a few warmups with this function are needed initially.
fn update_val(&mut self, c: u8) {
self.hash1.val = ((self.hash1.val << HASH_SHIFT) ^ c as u16) & HASH_MASK;
}
pub fn update(&mut self, array: &[u8], pos: usize) {
let hash_value = array.get(pos + ZOPFLI_MIN_MATCH - 1).cloned().unwrap_or(0);
self.update_val(hash_value);
let hpos = pos & ZOPFLI_WINDOW_MASK;
self.hash1.update(hpos);
// Update "same".
let mut amount = 0;
let same_index = pos.wrapping_sub(1) & ZOPFLI_WINDOW_MASK;
let same = self.same[same_index];
if same > 1 {
amount = same - 1;
}
self.same[hpos] = amount;
self.hash2.val = (amount.wrapping_sub(ZOPFLI_MIN_MATCH as u16) & 255) ^ self.hash1.val;
self.hash2.update(hpos);
}
pub fn prev_at(&self, index: usize, which: Which) -> usize {
(match which {
Which::Hash1 => self.hash1.prev_and_hashval[index].prev,
Which::Hash2 => self.hash2.prev_and_hashval[index].prev,
}) as usize
}
pub fn hash_val_at(&self, index: usize, which: Which) -> i32 {
let hashval = match which {
Which::Hash1 => self.hash1.prev_and_hashval[index].hashval,
Which::Hash2 => self.hash2.prev_and_hashval[index].hashval,
};
hashval.map_or(-1, |hv| hv as i32)
}
pub fn val(&self, which: Which) -> u16 {
match which {
Which::Hash1 => self.hash1.val,
Which::Hash2 => self.hash2.val,
}
}
}
#[cfg(feature = "std")]
pub static HASH_POOL: Lazy>> =
Lazy::new(|| LinearObjectPool::new(ZopfliHash::new, |boxed| boxed.reset()));
#[cfg(not(feature = "std"))]
#[derive(Copy, Clone)]
pub struct ZopfliHashFactory;
#[cfg(not(feature = "std"))]
impl ZopfliHashFactory {
pub fn pull(self) -> Box {
ZopfliHash::new()
}
}
#[cfg(not(feature = "std"))]
pub static HASH_POOL: ZopfliHashFactory = ZopfliHashFactory;
zopfli-0.8.1/src/io.rs 0000644 0000000 0000000 00000014736 10461020230 0012714 0 ustar 0000000 0000000 use alloc::vec::Vec;
use core::{
cmp, fmt,
fmt::{Display, Formatter},
mem,
};
/// A trait for objects which are byte-oriented sinks, modeled after its
/// `std::io::Write` counterpart.
///
/// The documentation for this trait is taken from the aforementioned `std` trait.
pub trait Write {
/// Write a buffer into this writer, returning how many bytes were written.
///
/// This function will attempt to write the entire contents of `buf`, but
/// the entire write might not succeed, or the write may also generate an
/// error. A call to `write` represents *at most one* attempt to write to
/// any wrapped object.
///
/// Calls to `write` are not guaranteed to block waiting for data to be
/// written, and a write which would otherwise block can be indicated through
/// an [`Err`] variant.
///
/// If the return value is `Ok(n)` then it must be guaranteed that
/// `n <= buf.len()`. A return value of `0` typically means that the
/// underlying object is no longer able to accept bytes and will likely not
/// be able to in the future as well, or that the buffer provided is empty.
///
/// # Errors
///
/// Each call to `write` may generate an I/O error indicating that the
/// operation could not be completed. If an error is returned then no bytes
/// in the buffer were written to this writer.
///
/// It is **not** considered an error if the entire buffer could not be
/// written to this writer.
///
/// An error of the [`ErrorKind::Interrupted`] kind is non-fatal and the
/// write operation should be retried if there is nothing else to do.
fn write(&mut self, buf: &[u8]) -> Result;
/// Flush this output stream, ensuring that all intermediately buffered
/// contents reach their destination.
///
/// # Errors
///
/// It is considered an error if not all bytes could be written due to
/// I/O errors or EOF being reached.
fn flush(&mut self) -> Result<(), Error>;
/// Attempts to write an entire buffer into this writer.
///
/// This method will continuously call [`write`] until there is no more data
/// to be written or an error of non-[`ErrorKind::Interrupted`] kind is
/// returned. This method will not return until the entire buffer has been
/// successfully written or such an error occurs. The first error that is
/// not of [`ErrorKind::Interrupted`] kind generated from this method will be
/// returned.
///
/// If the buffer contains no data, this will never call [`write`].
///
/// # Errors
///
/// This function will return the first error of
/// non-[`ErrorKind::Interrupted`] kind that [`write`] returns.
///
/// [`write`]: Write::write
// Implementation taken from Rust's stdlib
fn write_all(&mut self, mut buf: &[u8]) -> Result<(), Error> {
while !buf.is_empty() {
match self.write(buf) {
Ok(0) => {
return Err(ErrorKind::WriteZero.into());
}
Ok(n) => buf = &buf[n..],
Err(ref e) if e.kind() == ErrorKind::Interrupted => {}
Err(e) => return Err(e),
}
}
Ok(())
}
/// Creates a "by reference" adapter for this instance of `Write`.
///
/// The returned adapter also implements `Write` and will simply borrow this
/// current writer.
// Implementation taken from Rust's stdlib
fn by_ref(&mut self) -> &mut Self
where
Self: Sized,
{
self
}
}
impl Write for &mut [u8] {
// Implementation taken from Rust's stdlib
#[inline]
fn write(&mut self, data: &[u8]) -> Result {
let amt = cmp::min(data.len(), self.len());
let (a, b) = mem::take(self).split_at_mut(amt);
a.copy_from_slice(&data[..amt]);
*self = b;
Ok(amt)
}
#[inline]
fn flush(&mut self) -> Result<(), Error> {
Ok(())
}
// Implementation taken from Rust's stdlib
#[inline]
fn write_all(&mut self, data: &[u8]) -> Result<(), Error> {
if self.write(data)? == data.len() {
Ok(())
} else {
Err(ErrorKind::WriteZero.into())
}
}
}
impl Write for Vec {
// Implementation taken from Rust's stdlib
#[inline]
fn write(&mut self, buf: &[u8]) -> Result {
self.extend_from_slice(buf);
Ok(buf.len())
}
#[inline]
fn flush(&mut self) -> Result<(), Error> {
Ok(())
}
// Implementation taken from Rust's stdlib
#[inline]
fn write_all(&mut self, buf: &[u8]) -> Result<(), Error> {
self.extend_from_slice(buf);
Ok(())
}
}
impl Write for &mut W {
fn write(&mut self, buf: &[u8]) -> Result {
(**self).write(buf)
}
fn flush(&mut self) -> Result<(), Error> {
(**self).flush()
}
fn write_all(&mut self, buf: &[u8]) -> Result<(), Error> {
(**self).write_all(buf)
}
}
/// The error type for I/O operations of the `Write` trait.
#[derive(Debug)]
pub struct Error {
kind: ErrorKind,
}
impl Error {
/// Returns the corresponding `ErrorKind` for this error.
pub fn kind(&self) -> ErrorKind {
self.kind
}
}
impl Display for Error {
fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
f.write_fmt(format_args!("{:?}", self.kind))
}
}
impl From for Error {
fn from(value: ErrorKind) -> Self {
Self { kind: value }
}
}
#[cfg(feature = "nightly")]
impl core::error::Error for Error {}
/// A list specifying general categories of I/O error.
///
/// This list is intended to grow over time and it is not recommended to
/// exhaustively match against it.
///
/// The documentation for this enum is taken from the `std::io::ErrorKind`.
#[derive(Debug, Copy, Clone, Eq, PartialEq)]
#[non_exhaustive]
pub enum ErrorKind {
/// This operation was interrupted.
///
/// Interrupted operations can typically be retried.
Interrupted,
/// An error returned when an operation could not be completed because a
/// call to [`write`] returned [`Ok(0)`].
///
/// This typically means that an operation could only succeed if it wrote a
/// particular number of bytes but only a smaller number of bytes could be
/// written.
///
/// [`write`]: Write::write
/// [`Ok(0)`]: Ok
WriteZero,
/// An error that does not fall under any other I/O error kind.
Other,
}
zopfli-0.8.1/src/iter.rs 0000644 0000000 0000000 00000001076 10461020230 0013241 0 ustar 0000000 0000000 use core::iter::Peekable;
pub struct FlagLastIterator {
inner: Peekable,
}
impl Iterator for FlagLastIterator {
type Item = (I::Item, bool);
fn next(&mut self) -> Option {
self.inner
.next()
.map(|val| (val, self.inner.peek().is_none()))
}
}
pub trait ToFlagLastIterator: Iterator + Sized {
fn flag_last(self) -> FlagLastIterator {
FlagLastIterator {
inner: self.peekable(),
}
}
}
impl ToFlagLastIterator for T {}
zopfli-0.8.1/src/katajainen.rs 0000644 0000000 0000000 00000027101 10461020230 0014400 0 ustar 0000000 0000000 use alloc::vec::Vec;
use core::{
cell::Cell,
cmp::{self, Ordering},
};
use bumpalo::Bump;
// Bounded package merge algorithm, based on the paper
// "A Fast and Space-Economical Algorithm for Length-Limited Coding
// Jyrki Katajainen, Alistair Moffat, Andrew Turpin".
struct Thing<'a> {
node_arena: &'a Bump,
leaves: Vec,
lists: [List<'a>; 15],
}
struct Node<'a> {
weight: usize,
count: usize,
tail: Cell>>,
}
struct Leaf {
weight: usize,
count: usize,
}
impl PartialEq for Leaf {
fn eq(&self, other: &Self) -> bool {
self.weight == other.weight
}
}
impl Eq for Leaf {}
impl Ord for Leaf {
fn cmp(&self, other: &Self) -> Ordering {
self.weight.cmp(&other.weight)
}
}
impl PartialOrd for Leaf {
fn partial_cmp(&self, other: &Self) -> Option {
Some(self.cmp(other))
}
}
#[derive(Clone, Copy)]
struct List<'arena> {
lookahead0: &'arena Node<'arena>,
lookahead1: &'arena Node<'arena>,
}
/// Calculates the bitlengths for the Huffman tree, based on the counts of each
/// symbol.
pub fn length_limited_code_lengths(frequencies: &[usize], max_bits: usize) -> Vec {
let num_freqs = frequencies.len();
assert!(num_freqs <= 288);
// Count used symbols and place them in the leaves.
let mut leaves = frequencies
.iter()
.enumerate()
.filter_map(|(i, &freq)| {
(freq != 0).then_some(Leaf {
weight: freq,
count: i,
})
})
.collect::>();
let num_symbols = leaves.len();
// Short circuit some special cases
// TODO:
// if ((1 << maxbits) < numsymbols) {
// free(leaves);
// return 1; /* Error, too few maxbits to represent symbols. */
// }
if num_symbols <= 2 {
// The symbols for the non-zero frequencies can be represented
// with zero or one bits.
let mut bit_lengths = vec![0; num_freqs];
for i in 0..num_symbols {
bit_lengths[leaves[i].count] = 1;
}
return bit_lengths;
}
// Sort the leaves from least frequent to most frequent.
leaves.sort();
let max_bits = cmp::min(num_symbols - 1, max_bits);
assert!(max_bits <= 15);
let arena_capacity = max_bits * 2 * num_symbols;
let node_arena = Bump::with_capacity(arena_capacity);
let node0 = node_arena.alloc(Node {
weight: leaves[0].weight,
count: 1,
tail: Cell::new(None),
});
let node1 = node_arena.alloc(Node {
weight: leaves[1].weight,
count: 2,
tail: Cell::new(None),
});
let lists = [List {
lookahead0: node0,
lookahead1: node1,
}; 15];
let mut thing = Thing {
node_arena: &node_arena,
leaves,
lists,
};
// In the last list, 2 * num_symbols - 2 active chains need to be created. Two
// are already created in the initialization. Each boundary_pm run creates one.
let num_boundary_pm_runs = 2 * num_symbols - 4;
for _ in 0..num_boundary_pm_runs - 1 {
thing.boundary_pm(max_bits - 1);
}
thing.boundary_pm_final(max_bits - 1);
thing.extract_bit_lengths(max_bits, num_freqs)
}
impl<'a> Thing<'a> {
fn boundary_pm(&mut self, index: usize) {
let num_symbols = self.leaves.len();
let last_count = self.lists[index].lookahead1.count; // Count of last chain of list.
if index == 0 && last_count >= num_symbols {
return;
}
self.lists[index].lookahead0 = self.lists[index].lookahead1;
if index == 0 {
// New leaf node in list 0.
let new_chain = self.node_arena.alloc(Node {
weight: self.leaves[last_count].weight,
count: last_count + 1,
tail: self.lists[index].lookahead0.tail.clone(),
});
self.lists[index].lookahead1 = new_chain;
} else {
let weight_sum = {
let previous_list = &self.lists[index - 1];
previous_list.lookahead0.weight + previous_list.lookahead1.weight
};
if last_count < num_symbols && weight_sum > self.leaves[last_count].weight {
// New leaf inserted in list, so count is incremented.
let new_chain = self.node_arena.alloc(Node {
weight: self.leaves[last_count].weight,
count: last_count + 1,
tail: self.lists[index].lookahead0.tail.clone(),
});
self.lists[index].lookahead1 = new_chain;
} else {
let new_chain = self.node_arena.alloc(Node {
weight: weight_sum,
count: last_count,
tail: Cell::new(Some(self.lists[index - 1].lookahead1)),
});
self.lists[index].lookahead1 = new_chain;
// Two lookahead chains of previous list used up, create new ones.
self.boundary_pm(index - 1);
self.boundary_pm(index - 1);
}
}
}
fn boundary_pm_final(&mut self, index: usize) {
let num_symbols = self.leaves.len();
// Count of last chain of list.
let last_count = self.lists[index].lookahead1.count;
let weight_sum = {
let previous_list = &self.lists[index - 1];
previous_list.lookahead0.weight + previous_list.lookahead1.weight
};
if last_count < num_symbols && weight_sum > self.leaves[last_count].weight {
let new_chain = self.node_arena.alloc(Node {
weight: 0,
count: last_count + 1,
tail: self.lists[index].lookahead1.tail.clone(),
});
self.lists[index].lookahead1 = new_chain;
} else {
self.lists[index]
.lookahead1
.tail
.set(Some(self.lists[index - 1].lookahead1));
}
}
fn extract_bit_lengths(&self, max_bits: usize, num_freqs: usize) -> Vec {
let mut counts = [0; 16];
let mut end = 16;
let mut ptr = 15;
let mut value = 1;
let mut node = self.lists[max_bits - 1].lookahead1;
end -= 1;
counts[end] = node.count;
while let Some(tail) = node.tail.get() {
end -= 1;
counts[end] = tail.count;
node = tail;
}
let mut val = counts[15];
let mut bit_lengths = vec![0; num_freqs];
while ptr >= end {
while val > counts[ptr - 1] {
bit_lengths[self.leaves[val - 1].count] = value;
val -= 1;
}
ptr -= 1;
value += 1;
}
bit_lengths
}
}
// fn next_leaf(lists: &mut [List], leaves: &[Leaf], current_list_index: usize) {
// let mut current_list = &mut lists[current_list_index];
//
// // The next leaf goes next; counting itself makes the leaf_count increase by one.
// current_list.lookahead1.weight = leaves[current_list.next_leaf_index].weight;
// current_list.lookahead1.leaf_counts.clear();
// current_list.lookahead1.leaf_counts.extend_from_slice(¤t_list.lookahead0.leaf_counts);
// let last_index = current_list.lookahead1.leaf_counts.len() - 1;
// current_list.lookahead1.leaf_counts[last_index] += 1;
// current_list.next_leaf_index += 1;
// }
// fn next_tree(weight_sum: usize, lists: &mut [List], leaves: &[Leaf], current_list_index: usize) {
// {
// let (head, tail) = lists.split_at_mut(current_list_index);
// let prev_list = head.last_mut().unwrap();
// let current_list = tail.first_mut().unwrap();
//
// let previous_list_leaf_counts = &prev_list.lookahead1.leaf_counts;
//
// // Make a tree from the lookaheads from the previous list; that goes next.
// // This is not a leaf node, so the leaf count stays the same.
// current_list.lookahead1.weight = weight_sum;
// current_list.lookahead1.leaf_counts.clear();
//
// current_list.lookahead1.leaf_counts.extend_from_slice(previous_list_leaf_counts);
// current_list.lookahead1.leaf_counts.push(*current_list.lookahead0.leaf_counts.last().unwrap());
// }
//
// // The previous list needs two new lookahead nodes.
// boundary_pm(lists, leaves, current_list_index - 1);
// boundary_pm(lists, leaves, current_list_index - 1);
// }
// fn boundary_pm_toplevel(lists: &mut [List], leaves: &[Leaf]) {
// let last_index = lists.len() - 1;
// boundary_pm(lists, leaves, last_index);
// }
// fn boundary_pm(lists: &mut [List], leaves: &[Leaf], current_list_index: usize) {
// let next_leaf_index = lists[current_list_index].next_leaf_index;
// let num_symbols = leaves.len();
//
// if current_list_index == 0 && next_leaf_index == num_symbols {
// // We've added all the leaves to the lowest list, so we're done here
// return;
// }
//
// mem::swap(&mut lists[current_list_index].lookahead0, &mut lists[current_list_index].lookahead1);
//
// if current_list_index == 0 {
// lowest_list(lists, leaves);
// } else {
// // We're at a list other than the lowest list.
// let weight_sum = {
// let previous_list = &lists[current_list_index - 1];
// previous_list.lookahead0.weight + previous_list.lookahead1.weight
// };
//
// if next_leaf_index < num_symbols && weight_sum > leaves[next_leaf_index].weight {
// next_leaf(lists, leaves, current_list_index);
// } else {
// next_tree(weight_sum, lists, leaves, current_list_index);
// }
// }
// }
#[cfg(test)]
mod test {
use super::*;
#[test]
fn test_from_paper_3() {
let input = [1, 1, 5, 7, 10, 14];
let output = length_limited_code_lengths(&input, 3);
let answer = vec![3, 3, 3, 3, 2, 2];
assert_eq!(output, answer);
}
#[test]
fn test_from_paper_4() {
let input = [1, 1, 5, 7, 10, 14];
let output = length_limited_code_lengths(&input, 4);
let answer = vec![4, 4, 3, 2, 2, 2];
assert_eq!(output, answer);
}
#[test]
fn max_bits_7() {
let input = [252, 0, 1, 6, 9, 10, 6, 3, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
let output = length_limited_code_lengths(&input, 7);
let answer = vec![1, 0, 6, 4, 3, 3, 3, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
assert_eq!(output, answer);
}
#[test]
fn max_bits_15() {
let input = [
0, 0, 0, 0, 0, 0, 18, 0, 6, 0, 12, 2, 14, 9, 27, 15, 23, 15, 17, 8, 1, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0,
];
let output = length_limited_code_lengths(&input, 15);
let answer = vec![
0, 0, 0, 0, 0, 0, 3, 0, 5, 0, 4, 6, 4, 4, 3, 4, 3, 3, 3, 4, 6, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0,
];
assert_eq!(output, answer);
}
#[test]
fn no_frequencies() {
let input = [0, 0, 0, 0, 0];
let output = length_limited_code_lengths(&input, 7);
let answer = vec![0, 0, 0, 0, 0];
assert_eq!(output, answer);
}
#[test]
fn only_one_frequency() {
let input = [0, 10, 0];
let output = length_limited_code_lengths(&input, 7);
let answer = vec![0, 1, 0];
assert_eq!(output, answer);
}
#[test]
fn only_two_frequencies() {
let input = [0, 0, 0, 0, 252, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
let output = length_limited_code_lengths(&input, 7);
let answer = [0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];
assert_eq!(output, answer);
}
}
zopfli-0.8.1/src/lib.rs 0000644 0000000 0000000 00000017522 10461020230 0013047 0 ustar 0000000 0000000 #![deny(trivial_casts, trivial_numeric_casts, missing_docs)]
//! A reimplementation of the [Zopfli](https://github.com/google/zopfli) compression library in Rust.
//!
//! Zopfli is a state of the art DEFLATE compressor that heavily prioritizes compression over speed.
//! It usually compresses much better than other DEFLATE compressors, generating standard DEFLATE
//! streams that can be decompressed with any DEFLATE decompressor, at the cost of being
//! significantly slower.
//!
//! # Features
//!
//! This crate exposes the following features. You can enable or disable them in your `Cargo.toml`
//! as needed.
//!
//! - `gzip` (enabled by default): enables support for compression in the gzip format.
//! - `zlib` (enabled by default): enables support for compression in the Zlib format.
//! - `std` (enabled by default): enables linking against the Rust standard library. When not enabled,
//! the crate is built with the `#![no_std]` attribute and can be used
//! in any environment where [`alloc`](https://doc.rust-lang.org/alloc/)
//! (i.e., a memory allocator) is available. In addition, the crate
//! exposes minimalist versions of the `std` I/O traits it needs to
//! function, allowing users to implement them. Disabling `std` requires
//! enabling `nightly` due to dependencies on unstable language features.
//! - `nightly`: enables performance optimizations that are specific to the nightly Rust toolchain.
//! Currently, this feature improves rustdoc generation and enables the namesake feature
//! on `crc32fast`, but this may change in the future. This feature also used to enable
//! `simd-adler32`'s namesake feature, but it no longer does as the latest `simd-adler32`
//! release does not build with the latest nightlies (as of 2024-05-18) when that feature
//! is enabled.
#![cfg_attr(not(feature = "std"), no_std)]
#![cfg_attr(feature = "nightly", feature(doc_auto_cfg), feature(error_in_core))]
// No-op log implementation for no-std targets
#[cfg(not(feature = "std"))]
macro_rules! debug {
( $( $_:expr ),* ) => {};
}
#[cfg(not(feature = "std"))]
macro_rules! trace {
( $( $_:expr ),* ) => {};
}
#[cfg(not(feature = "std"))]
macro_rules! log_enabled {
( $( $_:expr ),* ) => {
false
};
}
#[cfg_attr(not(feature = "std"), macro_use)]
extern crate alloc;
pub use deflate::{BlockType, DeflateEncoder};
#[cfg(feature = "gzip")]
pub use gzip::GzipEncoder;
#[cfg(all(test, feature = "std"))]
use proptest::prelude::*;
#[cfg(feature = "zlib")]
pub use zlib::ZlibEncoder;
mod blocksplitter;
mod cache;
mod deflate;
#[cfg(feature = "gzip")]
mod gzip;
mod hash;
#[cfg(any(doc, not(feature = "std")))]
mod io;
mod iter;
mod katajainen;
mod lz77;
#[cfg(not(feature = "std"))]
mod math;
mod squeeze;
mod symbols;
mod tree;
mod util;
#[cfg(feature = "zlib")]
mod zlib;
use core::num::NonZeroU64;
#[cfg(all(not(doc), feature = "std"))]
use std::io::{Error, Write};
#[cfg(any(doc, not(feature = "std")))]
pub use io::{Error, ErrorKind, Write};
/// Options for the Zopfli compression algorithm.
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
#[cfg_attr(all(test, feature = "std"), derive(proptest_derive::Arbitrary))]
pub struct Options {
/// Maximum amount of times to rerun forward and backward pass to optimize LZ77
/// compression cost.
/// Good values: 10, 15 for small files, 5 for files over several MB in size or
/// it will be too slow.
///
/// Default value: 15.
#[cfg_attr(
all(test, feature = "std"),
proptest(
strategy = "(1..=10u64).prop_map(|iteration_count| NonZeroU64::new(iteration_count).unwrap())"
)
)]
pub iteration_count: NonZeroU64,
/// Stop after rerunning forward and backward pass this many times without finding
/// a smaller representation of the block.
///
/// Default value: practically infinite (maximum `u64` value)
pub iterations_without_improvement: NonZeroU64,
/// Maximum amount of blocks to split into (0 for unlimited, but this can give
/// extreme results that hurt compression on some files).
///
/// Default value: 15.
pub maximum_block_splits: u16,
}
impl Default for Options {
fn default() -> Options {
Options {
iteration_count: NonZeroU64::new(15).unwrap(),
iterations_without_improvement: NonZeroU64::new(u64::MAX).unwrap(),
maximum_block_splits: 15,
}
}
}
/// The output file format to use to store data compressed with Zopfli.
#[derive(Debug, Copy, Clone)]
#[cfg(feature = "std")]
pub enum Format {
/// The gzip file format, as defined in
/// [RFC 1952](https://datatracker.ietf.org/doc/html/rfc1952).
///
/// This file format can be easily decompressed with the gzip
/// program.
#[cfg(feature = "gzip")]
Gzip,
/// The zlib file format, as defined in
/// [RFC 1950](https://datatracker.ietf.org/doc/html/rfc1950).
///
/// The zlib format has less header overhead than gzip, but it
/// stores less metadata.
#[cfg(feature = "zlib")]
Zlib,
/// The raw DEFLATE stream format, as defined in
/// [RFC 1951](https://datatracker.ietf.org/doc/html/rfc1951).
///
/// Raw DEFLATE streams are not meant to be stored as-is because
/// they lack error detection and correction metadata. They
/// are usually embedded in other file formats, such as gzip
/// and zlib.
Deflate,
}
/// Compresses data from a source with the Zopfli algorithm, using the specified
/// options, and writes the result to a sink in the defined output format.
#[cfg(feature = "std")]
pub fn compress(
options: Options,
output_format: Format,
mut in_data: R,
out: W,
) -> Result<(), Error> {
match output_format {
#[cfg(feature = "gzip")]
Format::Gzip => {
let mut gzip_encoder = GzipEncoder::new_buffered(options, BlockType::Dynamic, out)?;
std::io::copy(&mut in_data, &mut gzip_encoder)?;
gzip_encoder.into_inner()?.finish().map(|_| ())
}
#[cfg(feature = "zlib")]
Format::Zlib => {
let mut zlib_encoder = ZlibEncoder::new_buffered(options, BlockType::Dynamic, out)?;
std::io::copy(&mut in_data, &mut zlib_encoder)?;
zlib_encoder.into_inner()?.finish().map(|_| ())
}
Format::Deflate => {
let mut deflate_encoder =
DeflateEncoder::new_buffered(options, BlockType::Dynamic, out);
std::io::copy(&mut in_data, &mut deflate_encoder)?;
deflate_encoder.into_inner()?.finish().map(|_| ())
}
}
}
/// Populates object pools for expensive objects that Zopfli uses. Call this on a background thread
/// when you know ahead of time that compression will be needed.
#[cfg(feature = "std")]
pub fn prewarm_object_pools() {
hash::HASH_POOL.pull();
}
#[cfg(all(test, feature = "std"))]
mod test {
use std::io;
use miniz_oxide::inflate;
use proptest::proptest;
use super::*;
proptest! {
#[test]
fn deflating_is_reversible(
options: Options,
btype: BlockType,
data in prop::collection::vec(any::(), 0..64 * 1024)
) {
let mut compressed_data = Vec::with_capacity(data.len());
let mut encoder = DeflateEncoder::new(options, btype, &mut compressed_data);
io::copy(&mut &*data, &mut encoder).unwrap();
encoder.finish().unwrap();
let decompressed_data = inflate::decompress_to_vec(&compressed_data).expect("Could not inflate compressed stream");
prop_assert_eq!(data, decompressed_data, "Decompressed data should match input data");
}
}
}
zopfli-0.8.1/src/lz77.rs 0000644 0000000 0000000 00000046675 10461020230 0013117 0 ustar 0000000 0000000 use alloc::{boxed::Box, vec::Vec};
use core::cmp;
use crate::{
cache::Cache,
hash::{Which, ZopfliHash, HASH_POOL},
symbols::{get_dist_symbol, get_length_symbol},
util::{
boxed_array, ZOPFLI_MAX_CHAIN_HITS, ZOPFLI_MAX_MATCH, ZOPFLI_MIN_MATCH, ZOPFLI_NUM_D,
ZOPFLI_NUM_LL, ZOPFLI_WINDOW_MASK, ZOPFLI_WINDOW_SIZE,
},
};
#[derive(Clone, Copy)]
pub enum LitLen {
Literal(u16),
LengthDist(u16, u16),
}
impl LitLen {
pub fn size(&self) -> usize {
match *self {
LitLen::Literal(_) => 1,
LitLen::LengthDist(len, _) => len as usize,
}
}
}
/// Stores lit/length and dist pairs for LZ77.
/// Parameter litlens: Contains the literal symbols or length values.
/// Parameter dists: Contains the distances. A value is 0 to indicate that there is
/// no dist and the corresponding litlens value is a literal instead of a length.
/// Parameter size: The size of both the litlens and dists arrays.
#[derive(Clone, Default)]
pub struct Lz77Store {
pub litlens: Vec,
pub pos: Vec,
ll_symbol: Vec,
d_symbol: Vec,
ll_counts: Vec,
d_counts: Vec,
}
impl Lz77Store {
pub fn new() -> Lz77Store {
Lz77Store {
litlens: vec![],
pos: vec![],
ll_symbol: vec![],
d_symbol: vec![],
ll_counts: vec![],
d_counts: vec![],
}
}
pub fn reset(&mut self) {
self.litlens.clear();
self.pos.clear();
self.ll_symbol.clear();
self.d_symbol.clear();
self.ll_counts.clear();
self.d_counts.clear();
}
pub fn size(&self) -> usize {
self.litlens.len()
}
pub fn append_store_item(&mut self, litlen: LitLen, pos: usize) {
let origsize = self.litlens.len();
let llstart = ZOPFLI_NUM_LL * (origsize / ZOPFLI_NUM_LL);
let dstart = ZOPFLI_NUM_D * (origsize / ZOPFLI_NUM_D);
if origsize % ZOPFLI_NUM_LL == 0 {
if origsize == 0 {
self.ll_counts.resize(origsize + ZOPFLI_NUM_LL, 0);
} else {
// Append last histogram
self.ll_counts
.extend_from_within((origsize - ZOPFLI_NUM_LL)..origsize);
}
}
if origsize % ZOPFLI_NUM_D == 0 {
if origsize == 0 {
self.d_counts.resize(ZOPFLI_NUM_D, 0);
} else {
// Append last histogram
self.d_counts
.extend_from_within((origsize - ZOPFLI_NUM_D)..origsize);
}
}
self.pos.push(pos);
// Why isn't this at the beginning of this function?
// assert(length < 259);
self.litlens.push(litlen);
match litlen {
LitLen::Literal(length) => {
self.ll_symbol.push(length);
self.d_symbol.push(0);
self.ll_counts[llstart + length as usize] += 1;
}
LitLen::LengthDist(length, dist) => {
let len_sym = get_length_symbol(length as usize);
self.ll_symbol.push(len_sym as u16);
self.d_symbol.push(get_dist_symbol(dist) as u16);
self.ll_counts[llstart + len_sym] += 1;
self.d_counts[dstart + get_dist_symbol(dist)] += 1;
}
}
}
pub fn lit_len_dist(&mut self, length: u16, dist: u16, pos: usize) {
let litlen = if dist == 0 {
LitLen::Literal(length)
} else {
LitLen::LengthDist(length, dist)
};
self.append_store_item(litlen, pos);
}
/// Does LZ77 using an algorithm similar to gzip, with lazy matching, rather than
/// with the slow but better "squeeze" implementation.
/// The result is placed in the Lz77Store.
/// If instart is larger than 0, it uses values before instart as starting
/// dictionary.
pub fn greedy(&mut self, lmc: &mut C, in_data: &[u8], instart: usize, inend: usize) {
if instart == inend {
return;
}
let windowstart = instart.saturating_sub(ZOPFLI_WINDOW_SIZE);
let mut h = HASH_POOL.pull();
let arr = &in_data[..inend];
h.warmup(arr, windowstart, inend);
for i in windowstart..instart {
h.update(arr, i);
}
let mut i = instart;
let mut leng;
let mut dist;
let mut lengthscore;
/* Lazy matching. */
let mut prev_length = 0;
let mut prev_match = 0;
let mut prevlengthscore;
let mut match_available = false;
while i < inend {
h.update(arr, i);
let longest_match =
find_longest_match(lmc, &h, arr, i, inend, instart, ZOPFLI_MAX_MATCH, &mut None);
dist = longest_match.distance;
leng = longest_match.length;
lengthscore = get_length_score(leng as i32, dist as i32);
/* Lazy matching. */
prevlengthscore = get_length_score(prev_length as i32, prev_match as i32);
if match_available {
match_available = false;
if lengthscore > prevlengthscore + 1 {
self.lit_len_dist(arr[i - 1] as u16, 0, i - 1);
if (lengthscore as usize) >= ZOPFLI_MIN_MATCH
&& (leng as usize) < ZOPFLI_MAX_MATCH
{
match_available = true;
prev_length = leng as u32;
prev_match = dist as u32;
i += 1;
continue;
}
} else {
/* Add previous to output. */
leng = prev_length as u16;
dist = prev_match as u16;
/* Add to output. */
verify_len_dist(arr, i - 1, dist, leng);
self.lit_len_dist(leng, dist, i - 1);
for _ in 2..leng {
debug_assert!(i < inend);
i += 1;
h.update(arr, i);
}
i += 1;
continue;
}
} else if (lengthscore as usize) >= ZOPFLI_MIN_MATCH
&& (leng as usize) < ZOPFLI_MAX_MATCH
{
match_available = true;
prev_length = leng as u32;
prev_match = dist as u32;
i += 1;
continue;
}
/* End of lazy matching. */
/* Add to output. */
if (lengthscore as usize) >= ZOPFLI_MIN_MATCH {
verify_len_dist(arr, i, dist, leng);
self.lit_len_dist(leng, dist, i);
} else {
leng = 1;
self.lit_len_dist(arr[i] as u16, 0, i);
}
for _ in 1..leng {
debug_assert!(i < inend);
i += 1;
h.update(arr, i);
}
i += 1;
}
}
pub fn follow_path(
&mut self,
in_data: &[u8],
instart: usize,
inend: usize,
path: Vec,
lmc: &mut C,
) {
let windowstart = instart.saturating_sub(ZOPFLI_WINDOW_SIZE);
if instart == inend {
return;
}
let mut h = HASH_POOL.pull();
let arr = &in_data[..inend];
h.warmup(arr, windowstart, inend);
for i in windowstart..instart {
h.update(arr, i);
}
let mut pos = instart;
for item in path.into_iter().rev() {
let mut length = item;
debug_assert!(pos < inend);
h.update(arr, pos);
// Add to output.
if length >= ZOPFLI_MIN_MATCH as u16 {
// Get the distance by recalculating longest match. The found length
// should match the length from the path.
let longest_match = find_longest_match(
lmc,
&h,
arr,
pos,
inend,
instart,
length as usize,
&mut None,
);
let dist = longest_match.distance;
let dummy_length = longest_match.length;
debug_assert!(!(dummy_length != length && length > 2 && dummy_length > 2));
verify_len_dist(arr, pos, dist, length);
self.lit_len_dist(length, dist, pos);
} else {
length = 1;
self.lit_len_dist(arr[pos] as u16, 0, pos);
}
debug_assert!(pos + (length as usize) <= inend);
for j in 1..(length as usize) {
h.update(arr, pos + j);
}
pos += length as usize;
}
}
fn get_histogram_at(
&self,
lpos: usize,
) -> (Box<[usize; ZOPFLI_NUM_LL]>, Box<[usize; ZOPFLI_NUM_D]>) {
let mut ll = boxed_array(0);
let mut d = boxed_array(0);
/* The real histogram is created by using the histogram for this chunk, but
all superfluous values of this chunk subtracted. */
let llpos = ZOPFLI_NUM_LL * (lpos / ZOPFLI_NUM_LL);
let dpos = ZOPFLI_NUM_D * (lpos / ZOPFLI_NUM_D);
for (i, item) in ll.iter_mut().enumerate() {
*item = self.ll_counts[llpos + i];
}
let end = cmp::min(llpos + ZOPFLI_NUM_LL, self.size());
for i in (lpos + 1)..end {
ll[self.ll_symbol[i] as usize] -= 1;
}
for (i, item) in d.iter_mut().enumerate() {
*item = self.d_counts[dpos + i];
}
let end = cmp::min(dpos + ZOPFLI_NUM_D, self.size());
for i in (lpos + 1)..end {
if let LitLen::LengthDist(_, _) = self.litlens[i] {
d[self.d_symbol[i] as usize] -= 1;
}
}
(ll, d)
}
/// Gets the histogram of lit/len and dist symbols in the given range, using the
/// cumulative histograms, so faster than adding one by one for large range. Does
/// not add the one end symbol of value 256.
pub fn get_histogram(
&self,
lstart: usize,
lend: usize,
) -> (Box<[usize; ZOPFLI_NUM_LL]>, Box<[usize; ZOPFLI_NUM_D]>) {
if lstart + ZOPFLI_NUM_LL * 3 > lend {
let mut ll_counts = boxed_array(0);
let mut d_counts = boxed_array(0);
for i in lstart..lend {
ll_counts[self.ll_symbol[i] as usize] += 1;
if let LitLen::LengthDist(_, _) = self.litlens[i] {
d_counts[self.d_symbol[i] as usize] += 1;
}
}
(ll_counts, d_counts)
} else {
/* Subtract the cumulative histograms at the end and the start to get the
histogram for this range. */
let (ll, d) = self.get_histogram_at(lend - 1);
if lstart > 0 {
let (ll2, d2) = self.get_histogram_at(lstart - 1);
(
ll.iter()
.zip(ll2.iter())
.map(|(&ll_item1, &ll_item2)| ll_item1 - ll_item2)
.collect::>()
.try_into()
.unwrap(),
d.iter()
.zip(d2.iter())
.map(|(&d_item1, &d_item2)| d_item1 - d_item2)
.collect::>()
.try_into()
.unwrap(),
)
} else {
(ll, d)
}
}
}
pub fn get_byte_range(&self, lstart: usize, lend: usize) -> usize {
if lstart == lend {
return 0;
}
let l = lend - 1;
self.pos[l] + self.litlens[l].size() - self.pos[lstart]
}
}
pub struct LongestMatch {
pub distance: u16,
pub length: u16,
pub from_cache: bool,
pub limit: usize,
}
impl LongestMatch {
pub fn new(limit: usize) -> Self {
LongestMatch {
distance: 0,
length: 0,
from_cache: false,
limit,
}
}
}
/// Finds how long the match of `scan` and `match` is. Can be used to find how many
/// bytes starting from `scan`, and from `match`, are equal. Returns the last byte
/// after `scan`, which is still equal to the corresponding byte after `match`.
/// `scan` is the position to compare; `match` is the earlier position to compare.
/// `end` is the last possible byte, beyond which to stop looking.
/// `safe_end` is a few (8) bytes before end, for comparing multiple bytes at once.
fn get_match(array: &[u8], scan_offset: usize, match_offset: usize, end: usize) -> usize {
let mut scan_offset = scan_offset;
let mut match_offset = match_offset;
// /* 8 checks at once per array bounds check (usize is 64-bit). */
// // C code has other options if usize is not 64-bit, but this is all I'm supporting
// while scan_offset < safe_end && array[scan_offset] as *const u64 == array[match_offset] as *const u64 {
// scan_offset += 8;
// match_offset += 8;
// }
/* The remaining few bytes. */
while scan_offset != end && array[scan_offset] == array[match_offset] {
scan_offset += 1;
match_offset += 1;
}
scan_offset
}
#[allow(clippy::too_many_arguments)]
pub fn find_longest_match(
lmc: &mut C,
h: &ZopfliHash,
array: &[u8],
pos: usize,
size: usize,
blockstart: usize,
limit: usize,
sublen: &mut Option<&mut [u16]>,
) -> LongestMatch {
let mut longest_match = lmc.try_get(pos, limit, sublen, blockstart);
if longest_match.from_cache {
debug_assert!(pos + (longest_match.length as usize) <= size);
return longest_match;
}
let mut limit = longest_match.limit;
debug_assert!(limit <= ZOPFLI_MAX_MATCH);
debug_assert!(limit >= ZOPFLI_MIN_MATCH);
debug_assert!(pos < size);
if size - pos < ZOPFLI_MIN_MATCH {
/* The rest of the code assumes there are at least ZOPFLI_MIN_MATCH bytes to
try. */
longest_match.distance = 0;
longest_match.length = 0;
longest_match.from_cache = false;
longest_match.limit = 0;
return longest_match;
}
if pos + limit > size {
limit = size - pos;
}
let (bestdist, bestlength) = find_longest_match_loop(h, array, pos, size, limit, sublen);
lmc.store(pos, limit, sublen, bestdist, bestlength, blockstart);
debug_assert!(bestlength <= limit as u16);
debug_assert!(pos + bestlength as usize <= size);
longest_match.distance = bestdist;
longest_match.length = bestlength;
longest_match.from_cache = false;
longest_match.limit = limit;
longest_match
}
fn find_longest_match_loop(
h: &ZopfliHash,
array: &[u8],
pos: usize,
size: usize,
limit: usize,
sublen: &mut Option<&mut [u16]>,
) -> (u16, u16) {
let mut which_hash = Which::Hash1;
let hpos = pos & ZOPFLI_WINDOW_MASK;
let mut pp = hpos; /* During the whole loop, p == hprev[pp]. */
let mut p = h.prev_at(pp, which_hash);
let mut dist = if p < pp {
pp - p
} else {
ZOPFLI_WINDOW_SIZE - p + pp
};
let mut bestlength = 1;
let mut bestdist = 0;
let mut chain_counter = ZOPFLI_MAX_CHAIN_HITS; /* For quitting early. */
let arrayend = pos + limit;
let mut scan_offset;
let mut match_offset;
/* Go through all distances. */
while dist < ZOPFLI_WINDOW_SIZE && chain_counter > 0 {
let mut currentlength = 0;
debug_assert!(p < ZOPFLI_WINDOW_SIZE);
debug_assert_eq!(p, h.prev_at(pp, which_hash));
debug_assert_eq!(h.hash_val_at(p, which_hash), h.val(which_hash) as i32);
if dist > 0 {
debug_assert!(pos < size);
debug_assert!(dist <= pos);
scan_offset = pos;
match_offset = pos - dist;
/* Testing the byte at position bestlength first, goes slightly faster. */
if pos + bestlength >= size
|| array[scan_offset + bestlength] == array[match_offset + bestlength]
{
let same0 = h.same[pos & ZOPFLI_WINDOW_MASK];
if same0 > 2 && array[scan_offset] == array[match_offset] {
let same1 = h.same[(pos - dist) & ZOPFLI_WINDOW_MASK];
let same = cmp::min(cmp::min(same0, same1), limit as u16) as usize;
scan_offset += same;
match_offset += same;
}
scan_offset = get_match(array, scan_offset, match_offset, arrayend);
currentlength = scan_offset - pos; /* The found length. */
}
if currentlength > bestlength {
if let Some(ref mut subl) = *sublen {
for sublength in subl.iter_mut().take(currentlength + 1).skip(bestlength + 1) {
*sublength = dist as u16;
}
}
bestdist = dist;
bestlength = currentlength;
if currentlength >= limit {
break;
}
}
}
/* Switch to the other hash once this will be more efficient. */
if which_hash == Which::Hash1
&& bestlength >= h.same[hpos] as usize
&& h.val(Which::Hash2) as i32 == h.hash_val_at(p, Which::Hash2)
{
/* Now use the hash that encodes the length and first byte. */
which_hash = Which::Hash2;
}
pp = p;
p = h.prev_at(p, which_hash);
if p == pp {
break; /* Uninited prev value. */
}
dist += if p < pp {
pp - p
} else {
ZOPFLI_WINDOW_SIZE - p + pp
};
chain_counter -= 1;
}
(bestdist as u16, bestlength as u16)
}
/// Gets a score of the length given the distance. Typically, the score of the
/// length is the length itself, but if the distance is very long, decrease the
/// score of the length a bit to make up for the fact that long distances use large
/// amounts of extra bits.
///
/// This is not an accurate score, it is a heuristic only for the greedy LZ77
/// implementation. More accurate cost models are employed later. Making this
/// heuristic more accurate may hurt rather than improve compression.
///
/// The two direct uses of this heuristic are:
/// -avoid using a length of 3 in combination with a long distance. This only has
/// an effect if length == 3.
/// -make a slightly better choice between the two options of the lazy matching.
///
/// Indirectly, this affects:
/// -the block split points if the default of block splitting first is used, in a
/// rather unpredictable way
/// -the first zopfli run, so it affects the chance of the first run being closer
/// to the optimal output
fn get_length_score(length: i32, distance: i32) -> i32 {
// At 1024, the distance uses 9+ extra bits and this seems to be the sweet spot
// on tested files.
if distance > 1024 {
length - 1
} else {
length
}
}
#[cfg(debug_assertions)]
fn verify_len_dist(data: &[u8], pos: usize, dist: u16, length: u16) {
for i in 0..length {
let d1 = data[pos - (dist as usize) + (i as usize)];
let d2 = data[pos + (i as usize)];
if d1 != d2 {
assert_eq!(d1, d2);
break;
}
}
}
#[cfg(not(debug_assertions))]
fn verify_len_dist(_data: &[u8], _pos: usize, _dist: u16, _length: u16) {}
zopfli-0.8.1/src/main.rs 0000644 0000000 0000000 00000003725 10461020230 0013225 0 ustar 0000000 0000000 use std::{
env,
fs::File,
io::{self, prelude::*},
};
#[cfg(feature = "std")]
use log::info;
fn main() {
let options = zopfli::Options::default();
let output_type = zopfli::Format::Gzip;
// TODO: CLI arguments
// TODO: Allow specifying output to STDOUT
let extension = match output_type {
zopfli::Format::Gzip => ".gz",
zopfli::Format::Zlib => ".zlib",
zopfli::Format::Deflate => ".deflate",
};
for filename in env::args().skip(1) {
let file = File::open(&filename)
.unwrap_or_else(|why| panic!("couldn't open {}: {}", filename, why));
let filesize = file.metadata().map(|x| x.len()).unwrap() as usize;
let out_filename = format!("{}{}", filename, extension);
// Attempt to create the output file, panic if the output file could not be opened
let out_file = File::create(&out_filename)
.unwrap_or_else(|why| panic!("couldn't create output file {}: {}", out_filename, why));
let mut out_file = WriteStatistics::new(out_file);
zopfli::compress(options, output_type, &file, &mut out_file).unwrap_or_else(|why| {
panic!("couldn't write to output file {}: {}", out_filename, why)
});
let out_size = out_file.count;
info!(
"Original Size: {}, Compressed: {}, Compression: {}% Removed",
filesize,
out_size,
100.0 * (filesize - out_size) as f64 / filesize as f64
);
}
}
struct WriteStatistics {
inner: W,
count: usize,
}
impl WriteStatistics {
fn new(inner: W) -> Self {
WriteStatistics { inner, count: 0 }
}
}
impl Write for WriteStatistics {
fn write(&mut self, buf: &[u8]) -> io::Result {
let res = self.inner.write(buf);
if let Ok(size) = res {
self.count += size;
}
res
}
fn flush(&mut self) -> io::Result<()> {
self.inner.flush()
}
}
zopfli-0.8.1/src/math.rs 0000644 0000000 0000000 00000005567 10461020230 0013240 0 ustar 0000000 0000000 #[allow(dead_code)] // False positive
/// Provides math operations for doubles on `no_std` targets that are not available on `core`.
pub trait F64MathExt {
/// Computes the absolute value of `self`.
#[must_use = "method returns a new number and does not mutate the original value"]
fn abs(self) -> Self;
/// Returns the natural logarithm of the number.
///
/// This method is allowed to return different values for 0 and negative numbers
/// than its counterpart on `std` on some exotic platforms.
#[must_use = "method returns a new number and does not mutate the original value"]
fn ln(self) -> Self;
}
impl F64MathExt for f64 {
#[inline]
fn abs(self) -> Self {
f64::from_bits(self.to_bits() & ((1u64 << 63) - 1))
}
fn ln(self) -> Self {
// This implementation is not numerically precise at all, but is good enough
// for our purposes still: the core_intrinsics feature is discouraged, and
// on e.g. x86 platforms they compile down to a libm call anyway, which can
// be troublesome for freestanding targets
logf(self as f32) as f64
}
}
// Function taken from:
// https://github.com/rust-lang/libm/blob/a0a5bd85c913fe5f6c42bc11ef292e575f97ca6d/src/math/logf.rs
// See the link above for author and licensing information of this code snippet
#[allow(clippy::eq_op, clippy::excessive_precision)]
fn logf(mut x: f32) -> f32 {
const LN2_HI: f32 = core::f32::consts::LN_2; /* 0x3f317180 */
const LN2_LO: f32 = 9.0580006145e-06; /* 0x3717f7d1 */
/* |(log(1+s)-log(1-s))/s - Lg(s)| < 2**-34.24 (~[-4.95e-11, 4.97e-11]). */
const LG1: f32 = 0.66666662693; /* 0xaaaaaa.0p-24*/
const LG2: f32 = 0.40000972152; /* 0xccce13.0p-25 */
const LG3: f32 = 0.28498786688; /* 0x91e9ee.0p-25 */
const LG4: f32 = 0.24279078841; /* 0xf89e26.0p-26 */
let x1p25 = f32::from_bits(0x4c000000); // 0x1p25f === 2 ^ 25
let mut ix = x.to_bits();
let mut k = 0i32;
if (ix < 0x00800000) || ((ix >> 31) != 0) {
/* x < 2**-126 */
if ix << 1 == 0 {
return -1. / (x * x); /* log(+-0)=-inf */
}
if (ix >> 31) != 0 {
return (x - x) / 0.; /* log(-#) = NaN */
}
/* subnormal number, scale up x */
k -= 25;
x *= x1p25;
ix = x.to_bits();
} else if ix >= 0x7f800000 {
return x;
} else if ix == 0x3f800000 {
return 0.;
}
/* reduce x into [sqrt(2)/2, sqrt(2)] */
ix += 0x3f800000 - 0x3f3504f3;
k += ((ix >> 23) as i32) - 0x7f;
ix = (ix & 0x007fffff) + 0x3f3504f3;
x = f32::from_bits(ix);
let f = x - 1.;
let s = f / (2. + f);
let z = s * s;
let w = z * z;
let t1 = w * (LG2 + w * LG4);
let t2 = z * (LG1 + w * LG3);
let r = t2 + t1;
let hfsq = 0.5 * f * f;
let dk = k as f32;
s * (hfsq + r) + dk * LN2_LO - hfsq + f + dk * LN2_HI
}
zopfli-0.8.1/src/squeeze.rs 0000644 0000000 0000000 00000043663 10461020230 0013767 0 ustar 0000000 0000000 //! The squeeze functions do enhanced LZ77 compression by optimal parsing with a
//! cost model, rather than greedily choosing the longest length or using a single
//! step of lazy matching like regular implementations.
//!
//! Since the cost model is based on the Huffman tree that can only be calculated
//! after the LZ77 data is generated, there is a chicken and egg problem, and
//! multiple runs are done with updated cost models to converge to a better
//! solution.
use alloc::vec::Vec;
use core::cmp;
#[cfg(feature = "std")]
use log::{debug, trace};
use crate::{
cache::Cache,
deflate::{calculate_block_size, BlockType},
hash::{ZopfliHash, HASH_POOL},
lz77::{find_longest_match, LitLen, Lz77Store},
symbols::{get_dist_extra_bits, get_dist_symbol, get_length_extra_bits, get_length_symbol},
util::{ZOPFLI_MAX_MATCH, ZOPFLI_NUM_D, ZOPFLI_NUM_LL, ZOPFLI_WINDOW_MASK, ZOPFLI_WINDOW_SIZE},
};
const K_INV_LOG2: f64 = core::f64::consts::LOG2_E; // 1.0 / log(2.0)
#[cfg(not(feature = "std"))]
#[allow(unused_imports)] // False-positive
use crate::math::F64MathExt;
/// Cost model which should exactly match fixed tree.
fn get_cost_fixed(litlen: usize, dist: u16) -> f64 {
let result = if dist == 0 {
if litlen <= 143 {
8
} else {
9
}
} else {
let dbits = get_dist_extra_bits(dist);
let lbits = get_length_extra_bits(litlen);
let lsym = get_length_symbol(litlen);
// Every dist symbol has length 5.
7 + (lsym > 279) as usize + 5 + dbits + lbits
};
result as f64
}
/// Cost model based on symbol statistics.
fn get_cost_stat(litlen: usize, dist: u16, stats: &SymbolStats) -> f64 {
if dist == 0 {
stats.ll_symbols[litlen]
} else {
let lsym = get_length_symbol(litlen);
let lbits = get_length_extra_bits(litlen) as f64;
let dsym = get_dist_symbol(dist);
let dbits = get_dist_extra_bits(dist) as f64;
lbits + dbits + stats.ll_symbols[lsym] + stats.d_symbols[dsym]
}
}
#[derive(Default)]
struct RanState {
m_w: u32,
m_z: u32,
}
impl RanState {
fn new() -> RanState {
RanState { m_w: 1, m_z: 2 }
}
/// Get random number: "Multiply-With-Carry" generator of G. Marsaglia
fn random_marsaglia(&mut self) -> u32 {
self.m_z = 36969 * (self.m_z & 65535) + (self.m_z >> 16);
self.m_w = 18000 * (self.m_w & 65535) + (self.m_w >> 16);
(self.m_z << 16).wrapping_add(self.m_w) // 32-bit result.
}
}
#[derive(Copy, Clone)]
struct SymbolStats {
/* The literal and length symbols. */
litlens: [usize; ZOPFLI_NUM_LL],
/* The 32 unique dist symbols, not the 32768 possible dists. */
dists: [usize; ZOPFLI_NUM_D],
/* Length of each lit/len symbol in bits. */
ll_symbols: [f64; ZOPFLI_NUM_LL],
/* Length of each dist symbol in bits. */
d_symbols: [f64; ZOPFLI_NUM_D],
}
impl Default for SymbolStats {
fn default() -> SymbolStats {
SymbolStats {
litlens: [0; ZOPFLI_NUM_LL],
dists: [0; ZOPFLI_NUM_D],
ll_symbols: [0.0; ZOPFLI_NUM_LL],
d_symbols: [0.0; ZOPFLI_NUM_D],
}
}
}
impl SymbolStats {
fn randomize_stat_freqs(&mut self, state: &mut RanState) {
fn randomize_freqs(freqs: &mut [usize], state: &mut RanState) {
let n = freqs.len();
let mut i = 0;
let end = n;
while i < end {
if (state.random_marsaglia() >> 4) % 3 == 0 {
let index = state.random_marsaglia() as usize % n;
freqs[i] = freqs[index];
}
i += 1;
}
}
randomize_freqs(&mut self.litlens, state);
randomize_freqs(&mut self.dists, state);
self.litlens[256] = 1; // End symbol.
}
/// Calculates the entropy of each symbol, based on the counts of each symbol. The
/// result is similar to the result of length_limited_code_lengths, but with the
/// actual theoretical bit lengths according to the entropy. Since the resulting
/// values are fractional, they cannot be used to encode the tree specified by
/// DEFLATE.
fn calculate_entropy(&mut self) {
fn calculate_and_store_entropy(count: &[usize], bitlengths: &mut [f64]) {
let n = count.len();
let sum = count.iter().sum();
let log2sum = (if sum == 0 { n } else { sum } as f64).ln() * K_INV_LOG2;
for i in 0..n {
// When the count of the symbol is 0, but its cost is requested anyway, it
// means the symbol will appear at least once anyway, so give it the cost as if
// its count is 1.
if count[i] == 0 {
bitlengths[i] = log2sum;
} else {
bitlengths[i] = log2sum - (count[i] as f64).ln() * K_INV_LOG2;
}
// Depending on compiler and architecture, the above subtraction of two
// floating point numbers may give a negative result very close to zero
// instead of zero (e.g. -5.973954e-17 with gcc 4.1.2 on Ubuntu 11.4). Clamp
// it to zero. These floating point imprecisions do not affect the cost model
// significantly so this is ok.
if bitlengths[i] < 0.0 && bitlengths[i] > -1E-5 {
bitlengths[i] = 0.0;
}
debug_assert!(bitlengths[i] >= 0.0);
}
}
calculate_and_store_entropy(&self.litlens, &mut self.ll_symbols);
calculate_and_store_entropy(&self.dists, &mut self.d_symbols);
}
/// Appends the symbol statistics from the store.
fn get_statistics(&mut self, store: &Lz77Store) {
for &litlen in &store.litlens {
match litlen {
LitLen::Literal(lit) => self.litlens[lit as usize] += 1,
LitLen::LengthDist(len, dist) => {
self.litlens[get_length_symbol(len as usize)] += 1;
self.dists[get_dist_symbol(dist)] += 1;
}
}
}
self.litlens[256] = 1; /* End symbol. */
self.calculate_entropy();
}
fn clear_freqs(&mut self) {
self.litlens = [0; ZOPFLI_NUM_LL];
self.dists = [0; ZOPFLI_NUM_D];
}
}
fn add_weighed_stat_freqs(
stats1: &SymbolStats,
w1: f64,
stats2: &SymbolStats,
w2: f64,
) -> SymbolStats {
let mut result = SymbolStats::default();
for i in 0..ZOPFLI_NUM_LL {
result.litlens[i] =
(stats1.litlens[i] as f64 * w1 + stats2.litlens[i] as f64 * w2) as usize;
}
for i in 0..ZOPFLI_NUM_D {
result.dists[i] = (stats1.dists[i] as f64 * w1 + stats2.dists[i] as f64 * w2) as usize;
}
result.litlens[256] = 1; // End symbol.
result
}
/// Finds the minimum possible cost this cost model can return for valid length and
/// distance symbols.
fn get_cost_model_min_cost f64>(costmodel: F) -> f64 {
let mut bestlength = 0; // length that has lowest cost in the cost model
let mut bestdist = 0; // distance that has lowest cost in the cost model
// Table of distances that have a different distance symbol in the deflate
// specification. Each value is the first distance that has a new symbol. Only
// different symbols affect the cost model so only these need to be checked.
// See RFC 1951 section 3.2.5. Compressed blocks (length and distance codes).
const DSYMBOLS: [u16; 30] = [
1, 2, 3, 4, 5, 7, 9, 13, 17, 25, 33, 49, 65, 97, 129, 193, 257, 385, 513, 769, 1025, 1537,
2049, 3073, 4097, 6145, 8193, 12289, 16385, 24577,
];
let mut mincost = f64::INFINITY;
for i in 3..259 {
let c = costmodel(i, 1);
if c < mincost {
bestlength = i;
mincost = c;
}
}
mincost = f64::INFINITY;
for dsym in DSYMBOLS {
let c = costmodel(3, dsym);
if c < mincost {
bestdist = dsym;
mincost = c;
}
}
costmodel(bestlength, bestdist)
}
/// Performs the forward pass for "squeeze". Gets the most optimal length to reach
/// every byte from a previous byte, using cost calculations.
/// `s`: the `ZopfliBlockState`
/// `in_data`: the input data array
/// `instart`: where to start
/// `inend`: where to stop (not inclusive)
/// `costmodel`: function to calculate the cost of some lit/len/dist pair.
/// `length_array`: output array of size `(inend - instart)` which will receive the best
/// length to reach this byte from a previous byte.
/// returns the cost that was, according to the `costmodel`, needed to get to the end.
fn get_best_lengths f64, C: Cache>(
lmc: &mut C,
in_data: &[u8],
instart: usize,
inend: usize,
costmodel: F,
h: &mut ZopfliHash,
costs: &mut Vec,
) -> (f64, Vec) {
// Best cost to get here so far.
let blocksize = inend - instart;
let mut length_array = vec![0; blocksize + 1];
if instart == inend {
return (0.0, length_array);
}
let windowstart = instart.saturating_sub(ZOPFLI_WINDOW_SIZE);
h.reset();
let arr = &in_data[..inend];
h.warmup(arr, windowstart, inend);
for i in windowstart..instart {
h.update(arr, i);
}
costs.resize(blocksize + 1, 0.0);
for cost in costs.iter_mut().take(blocksize + 1).skip(1) {
*cost = f32::INFINITY;
}
costs[0] = 0.0; /* Because it's the start. */
let mut i = instart;
let mut leng;
let mut longest_match;
let mut sublen = vec![0; ZOPFLI_MAX_MATCH + 1];
let mincost = get_cost_model_min_cost(&costmodel);
while i < inend {
let mut j = i - instart; // Index in the costs array and length_array.
h.update(arr, i);
// If we're in a long repetition of the same character and have more than
// ZOPFLI_MAX_MATCH characters before and after our position.
if h.same[i & ZOPFLI_WINDOW_MASK] > ZOPFLI_MAX_MATCH as u16 * 2
&& i > instart + ZOPFLI_MAX_MATCH + 1
&& i + ZOPFLI_MAX_MATCH * 2 + 1 < inend
&& h.same[(i - ZOPFLI_MAX_MATCH) & ZOPFLI_WINDOW_MASK] > ZOPFLI_MAX_MATCH as u16
{
let symbolcost = costmodel(ZOPFLI_MAX_MATCH, 1);
// Set the length to reach each one to ZOPFLI_MAX_MATCH, and the cost to
// the cost corresponding to that length. Doing this, we skip
// ZOPFLI_MAX_MATCH values to avoid calling ZopfliFindLongestMatch.
for _ in 0..ZOPFLI_MAX_MATCH {
costs[j + ZOPFLI_MAX_MATCH] = costs[j] + symbolcost as f32;
length_array[j + ZOPFLI_MAX_MATCH] = ZOPFLI_MAX_MATCH as u16;
i += 1;
j += 1;
h.update(arr, i);
}
}
longest_match = find_longest_match(
lmc,
h,
arr,
i,
inend,
instart,
ZOPFLI_MAX_MATCH,
&mut Some(&mut sublen),
);
leng = longest_match.length;
// Literal.
if i < inend {
let new_cost = costmodel(arr[i] as usize, 0) + costs[j] as f64;
debug_assert!(new_cost >= 0.0);
if new_cost < costs[j + 1] as f64 {
costs[j + 1] = new_cost as f32;
length_array[j + 1] = 1;
}
}
// Lengths.
let kend = cmp::min(leng as usize, inend - i);
let mincostaddcostj = mincost + costs[j] as f64;
for (k, &sublength) in sublen.iter().enumerate().take(kend + 1).skip(3) {
// Calling the cost model is expensive, avoid this if we are already at
// the minimum possible cost that it can return.
if costs[j + k] as f64 <= mincostaddcostj {
continue;
}
let new_cost = costmodel(k, sublength) + costs[j] as f64;
debug_assert!(new_cost >= 0.0);
if new_cost < costs[j + k] as f64 {
debug_assert!(k <= ZOPFLI_MAX_MATCH);
costs[j + k] = new_cost as f32;
length_array[j + k] = k as u16;
}
}
i += 1;
}
debug_assert!(costs[blocksize] >= 0.0);
(costs[blocksize] as f64, length_array)
}
/// Calculates the optimal path of lz77 lengths to use, from the calculated
/// `length_array`. The `length_array` must contain the optimal length to reach that
/// byte. The path will be filled with the lengths to use, so its data size will be
/// the amount of lz77 symbols.
fn trace(size: usize, length_array: &[u16]) -> Vec {
let mut index = size;
if size == 0 {
return vec![];
}
let mut path = Vec::with_capacity(index);
while index > 0 {
let lai = length_array[index];
let laiu = lai as usize;
path.push(lai);
debug_assert!(laiu <= index);
debug_assert!(laiu <= ZOPFLI_MAX_MATCH);
debug_assert_ne!(lai, 0);
index -= laiu;
}
path
}
/// Does a single run for `lz77_optimal`. For good compression, repeated runs
/// with updated statistics should be performed.
/// `s`: the block state
/// `in_data`: the input data array
/// `instart`: where to start
/// `inend`: where to stop (not inclusive)
/// `length_array`: array of size `(inend - instart)` used to store lengths
/// `costmodel`: function to use as the cost model for this squeeze run
/// `store`: place to output the LZ77 data
/// returns the cost that was, according to the `costmodel`, needed to get to the end.
/// This is not the actual cost.
#[allow(clippy::too_many_arguments)] // Not feasible to refactor in a more readable way
fn lz77_optimal_run f64, C: Cache>(
lmc: &mut C,
in_data: &[u8],
instart: usize,
inend: usize,
costmodel: F,
store: &mut Lz77Store,
h: &mut ZopfliHash,
costs: &mut Vec,
) {
let (cost, length_array) = get_best_lengths(lmc, in_data, instart, inend, costmodel, h, costs);
let path = trace(inend - instart, &length_array);
store.follow_path(in_data, instart, inend, path, lmc);
debug_assert!(cost < f64::INFINITY);
}
/// Does the same as `lz77_optimal`, but optimized for the fixed tree of the
/// deflate standard.
/// The fixed tree never gives the best compression. But this gives the best
/// possible LZ77 encoding possible with the fixed tree.
/// This does not create or output any fixed tree, only LZ77 data optimized for
/// using with a fixed tree.
/// If `instart` is larger than `0`, it uses values before `instart` as starting
/// dictionary.
pub fn lz77_optimal_fixed(
lmc: &mut C,
in_data: &[u8],
instart: usize,
inend: usize,
store: &mut Lz77Store,
) {
let mut h = HASH_POOL.pull();
let mut costs = Vec::with_capacity(inend - instart);
lz77_optimal_run(
lmc,
in_data,
instart,
inend,
get_cost_fixed,
store,
&mut h,
&mut costs,
);
}
/// Calculates lit/len and dist pairs for given data.
/// If `instart` is larger than 0, it uses values before `instart` as starting
/// dictionary.
pub fn lz77_optimal(
lmc: &mut C,
in_data: &[u8],
instart: usize,
inend: usize,
max_iterations: u64,
max_iterations_without_improvement: u64,
) -> Lz77Store {
/* Dist to get to here with smallest cost. */
let mut currentstore = Lz77Store::new();
let mut outputstore = currentstore.clone();
/* Initial run. */
currentstore.greedy(lmc, in_data, instart, inend);
let mut stats = SymbolStats::default();
stats.get_statistics(¤tstore);
let mut h = HASH_POOL.pull();
let mut costs = Vec::with_capacity(inend - instart + 1);
let mut beststats = SymbolStats::default();
let mut bestcost = f64::INFINITY;
let mut lastcost = 0.0;
/* Try randomizing the costs a bit once the size stabilizes. */
let mut ran_state = RanState::new();
let mut lastrandomstep = u64::MAX;
/* Do regular deflate, then loop multiple shortest path runs, each time using
the statistics of the previous run. */
/* Repeat statistics with each time the cost model from the previous stat
run. */
let mut current_iteration: u64 = 0;
let mut iterations_without_improvement: u64 = 0;
loop {
currentstore.reset();
lz77_optimal_run(
lmc,
in_data,
instart,
inend,
|a, b| get_cost_stat(a, b, &stats),
&mut currentstore,
&mut h,
&mut costs,
);
let cost = calculate_block_size(¤tstore, 0, currentstore.size(), BlockType::Dynamic);
if cost < bestcost {
iterations_without_improvement = 0;
/* Copy to the output store. */
outputstore = currentstore.clone();
beststats = stats;
bestcost = cost;
debug!("Iteration {}: {} bit", current_iteration, cost);
} else {
iterations_without_improvement += 1;
trace!("Iteration {}: {} bit", current_iteration, cost);
if iterations_without_improvement >= max_iterations_without_improvement {
break;
}
}
current_iteration += 1;
if current_iteration >= max_iterations {
break;
}
let laststats = stats;
stats.clear_freqs();
stats.get_statistics(¤tstore);
if lastrandomstep != u64::MAX {
/* This makes it converge slower but better. Do it only once the
randomness kicks in so that if the user does few iterations, it gives a
better result sooner. */
stats = add_weighed_stat_freqs(&stats, 1.0, &laststats, 0.5);
stats.calculate_entropy();
}
if current_iteration > 5 && (cost - lastcost).abs() < f64::EPSILON {
stats = beststats;
stats.randomize_stat_freqs(&mut ran_state);
stats.calculate_entropy();
lastrandomstep = current_iteration;
}
lastcost = cost;
}
outputstore
}
zopfli-0.8.1/src/symbols.rs 0000644 0000000 0000000 00000014245 10461020230 0013770 0 ustar 0000000 0000000 const LENGTH_SYMBOL_TABLE: [usize; 259] = [
0, 0, 0, 257, 258, 259, 260, 261, 262, 263, 264, 265, 265, 266, 266, 267, 267, 268, 268, 269,
269, 269, 269, 270, 270, 270, 270, 271, 271, 271, 271, 272, 272, 272, 272, 273, 273, 273, 273,
273, 273, 273, 273, 274, 274, 274, 274, 274, 274, 274, 274, 275, 275, 275, 275, 275, 275, 275,
275, 276, 276, 276, 276, 276, 276, 276, 276, 277, 277, 277, 277, 277, 277, 277, 277, 277, 277,
277, 277, 277, 277, 277, 277, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278, 278,
278, 278, 278, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279, 279,
280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 281, 281, 281,
281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 281,
281, 281, 281, 281, 281, 281, 281, 281, 281, 281, 282, 282, 282, 282, 282, 282, 282, 282, 282,
282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282, 282,
282, 282, 282, 282, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283,
283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 283, 284, 284,
284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 284,
284, 284, 284, 284, 284, 284, 284, 284, 284, 284, 285,
];
/// Gets the symbol for the given length, cfr. the DEFLATE spec.
/// Returns symbol in range [257-285] (inclusive).
pub fn get_length_symbol(length: usize) -> usize {
LENGTH_SYMBOL_TABLE[length]
}
/// Gets the amount of extra bits for the given dist, cfr. the DEFLATE spec.
pub fn get_dist_extra_bits(dist: u16) -> usize {
(match dist {
0..=4 => 0,
5..=8 => 1,
9..=16 => 2,
17..=32 => 3,
33..=64 => 4,
65..=128 => 5,
129..=256 => 6,
257..=512 => 7,
513..=1024 => 8,
1025..=2048 => 9,
2049..=4096 => 10,
4097..=8192 => 11,
8193..=16384 => 12,
_ => 13,
}) as usize
}
/// Gets value of the extra bits for the given dist, cfr. the DEFLATE spec.
pub fn get_dist_extra_bits_value(dist: u16) -> u16 {
match dist {
0..=4 => 0,
5..=8 => (dist - 5) & 1,
9..=16 => (dist - 9) & 3,
17..=32 => (dist - 17) & 7,
33..=64 => (dist - 33) & 15,
65..=128 => (dist - 65) & 31,
129..=256 => (dist - 129) & 63,
257..=512 => (dist - 257) & 127,
513..=1024 => (dist - 513) & 255,
1025..=2048 => (dist - 1025) & 511,
2049..=4096 => (dist - 2049) & 1023,
4097..=8192 => (dist - 4097) & 2047,
8193..=16384 => (dist - 8193) & 4095,
_ => (dist - 16385) & 8191,
}
}
pub fn get_dist_symbol(dist: u16) -> usize {
(match dist {
0..=4 => dist - 1,
5..=6 => 4,
7..=8 => 5,
9..=12 => 6,
13..=16 => 7,
17..=24 => 8,
25..=32 => 9,
33..=48 => 10,
49..=64 => 11,
65..=96 => 12,
97..=128 => 13,
129..=192 => 14,
193..=256 => 15,
257..=384 => 16,
385..=512 => 17,
513..=768 => 18,
769..=1024 => 19,
1025..=1536 => 20,
1537..=2048 => 21,
2049..=3072 => 22,
3073..=4096 => 23,
4097..=6144 => 24,
6145..=8192 => 25,
8193..=12288 => 26,
12289..=16384 => 27,
16385..=24576 => 28,
_ => 29,
}) as usize
}
const LENGTH_EXTRA_BITS: [usize; 259] = [
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3,
3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5,
5, 5, 0,
];
/// Gets the amount of extra bits for the given length, cfr. the DEFLATE spec.
pub fn get_length_extra_bits(l: usize) -> usize {
LENGTH_EXTRA_BITS[l]
}
const LENGTH_EXTRA_BITS_VALUE: [u32; 259] = [
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0,
1, 2, 3, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4,
5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5,
6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 0, 1, 2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, 30, 31, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 0,
];
/// Gets value of the extra bits for the given length, cfr. the DEFLATE spec.
pub fn get_length_extra_bits_value(l: usize) -> u32 {
LENGTH_EXTRA_BITS_VALUE[l]
}
const LENGTH_SYMBOL_EXTRA_BITS_TABLE: [u32; 29] = [
0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4, 5, 5, 5, 5, 0,
];
/// Gets the amount of extra bits for the given length symbol.
pub fn get_length_symbol_extra_bits(s: usize) -> u32 {
LENGTH_SYMBOL_EXTRA_BITS_TABLE[s - 257]
}
const DIST_SYMBOL_EXTRA_BITS_TABLE: [u32; 30] = [
0, 0, 0, 0, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 13,
13,
];
/// Gets the amount of extra bits for the given distance symbol.
pub fn get_dist_symbol_extra_bits(s: usize) -> u32 {
DIST_SYMBOL_EXTRA_BITS_TABLE[s]
}
zopfli-0.8.1/src/tree.rs 0000644 0000000 0000000 00000002270 10461020230 0013232 0 ustar 0000000 0000000 use alloc::vec::Vec;
/// Converts a series of Huffman tree bitlengths, to the bit values of the symbols.
pub fn lengths_to_symbols(lengths: &[u32], max_bits: u32) -> Vec {
let mut bl_count = vec![0; (max_bits + 1) as usize];
let mut next_code = vec![0; (max_bits + 1) as usize];
let n = lengths.len();
let mut symbols = vec![0; n];
// 1) Count the number of codes for each code length. Let bl_count[N] be the
// number of codes of length N, N >= 1. */
for &length in lengths.iter() {
assert!(length <= max_bits);
bl_count[length as usize] += 1;
}
// 2) Find the numerical value of the smallest code for each code length.
let mut code = 0;
bl_count[0] = 0;
for bits in 1..(max_bits + 1) {
code = (code + bl_count[(bits - 1) as usize]) << 1;
next_code[bits as usize] = code;
}
// 3) Assign numerical values to all codes, using consecutive values for all
// codes of the same length with the base values determined at step 2.
for i in 0..n {
let len = lengths[i] as usize;
if len != 0 {
symbols[i] = next_code[len];
next_code[len] += 1;
}
}
symbols
}
zopfli-0.8.1/src/util.rs 0000644 0000000 0000000 00000004410 10461020230 0013246 0 ustar 0000000 0000000 use alloc::boxed::Box;
/// Number of distinct literal/length symbols in DEFLATE
pub const ZOPFLI_NUM_LL: usize = 288;
/// Number of distinct distance symbols in DEFLATE
pub const ZOPFLI_NUM_D: usize = 32;
/// The window size for deflate. Must be a power of two. This should be 32768, the
/// maximum possible by the deflate spec. Anything less hurts compression more than
/// speed.
pub const ZOPFLI_WINDOW_SIZE: usize = 32768;
/// A block structure of huge, non-smart, blocks to divide the input into, to allow
/// operating on huge files without exceeding memory, such as the 1GB wiki9 corpus.
/// The whole compression algorithm, including the smarter block splitting, will
/// be executed independently on each huge block.
/// Dividing into huge blocks hurts compression, but not much relative to the size.
/// This must be equal or greater than `ZOPFLI_WINDOW_SIZE`.
#[cfg(feature = "std")]
pub const ZOPFLI_MASTER_BLOCK_SIZE: usize = 1_000_000;
/// The window mask used to wrap indices into the window. This is why the
/// window size must be a power of two.
pub const ZOPFLI_WINDOW_MASK: usize = ZOPFLI_WINDOW_SIZE - 1;
/// Maximum length that can be encoded in deflate.
pub const ZOPFLI_MAX_MATCH: usize = 258;
/// Minimum length that can be encoded in deflate.
pub const ZOPFLI_MIN_MATCH: usize = 3;
/// For longest match cache. max 256. Uses huge amounts of memory but makes it
/// faster. Uses this many times three bytes per single byte of the input data.
/// This is so because longest match finding has to find the exact distance
/// that belongs to each length for the best lz77 strategy.
/// Good values: e.g. 5, 8.
pub const ZOPFLI_CACHE_LENGTH: usize = 8;
/// limit the max hash chain hits for this hash value. This has an effect only
/// on files where the hash value is the same very often. On these files, this
/// gives worse compression (the value should ideally be 32768, which is the
/// `ZOPFLI_WINDOW_SIZE`, while zlib uses 4096 even for best level), but makes it
/// faster on some specific files.
/// Good value: e.g. 8192.
pub const ZOPFLI_MAX_CHAIN_HITS: usize = 8192;
#[inline]
pub fn boxed_array(element: T) -> Box<[T; N]> {
match vec![element; N].into_boxed_slice().try_into() {
Ok(x) => x,
Err(_) => unreachable!(),
}
}
zopfli-0.8.1/src/zlib.rs 0000644 0000000 0000000 00000007117 10461020230 0013240 0 ustar 0000000 0000000 use crate::{BlockType, DeflateEncoder, Error, Options, Write};
/// A Zlib encoder powered by the Zopfli algorithm, that compresses data using
/// a [`DeflateEncoder`]. Most users will find using [`compress`](crate::compress)
/// easier and more performant.
///
/// The caveats about short writes in [`DeflateEncoder`]s carry over to `ZlibEncoder`s:
/// for best performance and compression, it is best to avoid them. One way to ensure
/// this is to use the [`new_buffered`](ZlibEncoder::new_buffered) method.
pub struct ZlibEncoder {
deflate_encoder: Option>,
adler_hasher: simd_adler32::Adler32,
}
impl ZlibEncoder {
/// Creates a new Zlib encoder that will operate according to the
/// specified options.
pub fn new(options: Options, btype: BlockType, mut sink: W) -> Result {
let cmf = 120; // CM 8, CINFO 7. See zlib spec.
let flevel = 3;
let fdict = 0;
let mut cmfflg: u16 = 256 * cmf + fdict * 32 + flevel * 64;
let fcheck = 31 - cmfflg % 31;
cmfflg += fcheck;
sink.write_all(&cmfflg.to_be_bytes())?;
Ok(Self {
deflate_encoder: Some(DeflateEncoder::new(options, btype, sink)),
adler_hasher: simd_adler32::Adler32::new(),
})
}
/// Creates a new Zlib encoder that operates according to the specified
/// options and is wrapped with a buffer to guarantee that data is
/// compressed in large chunks, which is necessary for decent performance
/// and good compression ratio.
#[cfg(feature = "std")]
pub fn new_buffered(
options: Options,
btype: BlockType,
sink: W,
) -> Result, Error> {
Ok(std::io::BufWriter::with_capacity(
crate::util::ZOPFLI_MASTER_BLOCK_SIZE,
Self::new(options, btype, sink)?,
))
}
/// Encodes any pending chunks of data and writes them to the sink,
/// consuming the encoder and returning the wrapped sink. The sink
/// will have received a complete Zlib stream when this method
/// returns.
///
/// The encoder is automatically [`finish`](Self::finish)ed when
/// dropped, but explicitly finishing it with this method allows
/// handling I/O errors.
pub fn finish(mut self) -> Result {
self._finish().map(|sink| sink.unwrap())
}
fn _finish(&mut self) -> Result, Error> {
if self.deflate_encoder.is_none() {
return Ok(None);
}
let mut sink = self.deflate_encoder.take().unwrap().finish()?;
sink.write_all(&self.adler_hasher.finish().to_be_bytes())?;
Ok(Some(sink))
}
}
impl Write for ZlibEncoder {
fn write(&mut self, buf: &[u8]) -> Result {
self.deflate_encoder
.as_mut()
.unwrap()
.write(buf)
.map(|bytes_written| {
self.adler_hasher.write(&buf[..bytes_written]);
bytes_written
})
}
fn flush(&mut self) -> Result<(), Error> {
self.deflate_encoder.as_mut().unwrap().flush()
}
}
impl Drop for ZlibEncoder {
fn drop(&mut self) {
self._finish().ok();
}
}
// Boilerplate to make latest Rustdoc happy: https://github.com/rust-lang/rust/issues/117796
#[cfg(all(doc, feature = "std"))]
impl std::io::Write for ZlibEncoder {
fn write(&mut self, _buf: &[u8]) -> std::io::Result {
unimplemented!()
}
fn flush(&mut self) -> std::io::Result<()> {
unimplemented!()
}
}
