cacache-13.0.0/.cargo_vcs_info.json0000644000000001360000000000100124640ustar { "git": { "sha1": "ff333178ef2536c36b5c9675b615e1305ab72d16" }, "path_in_vcs": "" }cacache-13.0.0/.github/FUNDING.yml000064400000000000000000000005551046102023000144360ustar 00000000000000# These are supported funding model platforms github: [zkat] patreon: # Replace with a single Patreon username open_collective: # Replace with a single Open Collective username ko_fi: # Replace with a single Ko-fi username tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel custom: # Replace with a single custom sponsorship URL cacache-13.0.0/.github/workflows/ci.yml000064400000000000000000000025551046102023000157760ustar 00000000000000name: CI on: [push, pull_request] env: RUSTFLAGS: -Dwarnings jobs: fmt_and_docs: name: Check fmt & build docs runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - name: Install Rust uses: actions-rs/toolchain@v1 with: profile: minimal toolchain: stable components: rustfmt override: true - name: rustfmt run: cargo fmt --all -- --check - name: docs run: cargo doc build_and_test: name: Build & Test runs-on: ${{ matrix.os }} strategy: matrix: rust: [1.70.0, stable] os: [ubuntu-latest, macOS-latest, windows-latest] steps: - uses: actions/checkout@v1 - name: Install Rust uses: actions-rs/toolchain@v1 with: profile: minimal toolchain: ${{ matrix.rust }} components: clippy override: true - name: Clippy run: cargo clippy -- -D warnings - name: Check (async-std) run: cargo check --features link_to - name: Run tests (async-std) run: cargo test --verbose --features link_to - name: Check (Tokio) run: cargo check --no-default-features --features tokio-runtime,link_to - name: Run unit tests (Tokio) run: cargo test --verbose --no-default-features --features tokio-runtime,link_to --lib cacache-13.0.0/.gitignore000064400000000000000000000000471046102023000132450ustar 00000000000000/target **/*.rs.bk /.vscode Cargo.lock cacache-13.0.0/CHANGELOG.md000064400000000000000000000454211046102023000130730ustar 00000000000000# `cacache` Release Changelog ## 13.0.0 (2024-02-15) ### Bug Fixes * **sync:** Added the feature flags for compilation without the async runtime. Fixes #64. (#65) ([ffa1ab72](https://github.com/zkat/cacache-rs/commit/ffa1ab7254e0b051e6d65fe16b71ab44be04c968)) * **BREAKING CHANGE**: this bumps the MSRV to 1.70.0 * **remove_fully:** Remove the key content when set `remove_fully` to `true` (#63) ([761d7155](https://github.com/zkat/cacache-rs/commit/761d71558a4b9f06907cb5ff43247fdbd158716b)) * **BREAKING CHANGE**: this is technically a fix, but maybe people didn't expect this. ## 12.0.0 (2023-10-07) ### Features * **remove_fully:** add option for remove cache entry (#54) ([34ee6798](https://github.com/zkat/cacache-rs/commit/34ee679816cad95f617fbed11ad93b5f248ab50f)) * **reflink:** Separate reflink behavior into their own functions (#58) ([cfdf4eed](https://github.com/zkat/cacache-rs/commit/cfdf4eed4d19138ea8937b12333db8280d137b3a)) * **BREAKING CHANGE**: some signatures for copy have changed, and copy no longer automatically reflinks ## 11.7.1 (2023-09-11) ### Bug Fixes * **sync-only:** remove futures dep when no async features are being used ([100609ab](https://github.com/zkat/cacache-rs/commit/100609abee0715a12f1bf5688aa6b462b2b8c713)) * **mmap:** fix unused method warning ([c735d83a](https://github.com/zkat/cacache-rs/commit/c735d83a316ebe2844803132fa81f6dcb9f28b79)) * **deps:** moved to reflink-copy (#55) ([6624d66c](https://github.com/zkat/cacache-rs/commit/6624d66c16d2aedf6d8369c71c627ff7d817f72a)) ## 11.7.0 (2023-09-11) ### Features * **sync-only:** add support for removing async runtime dependency altogether (#56) ([60622267](https://github.com/zkat/cacache-rs/commit/606222678931cd809af35c1a38a58933b5fc6d7e)) ## 11.6.0 (2023-05-21) ### Features * **mmap:** new feature for mmap-based optimization (#49) ([58de0b2d](https://github.com/zkat/cacache-rs/commit/58de0b2d1016c964d6f18f042f10e2a744415b08)) * **xxhash:** Add xxhash support and some utilities for making it easier to use (#51) ([41392e74](https://github.com/zkat/cacache-rs/commit/41392e7469b7e49b5b35aaacf77eb8a877a1484c)) ### Bug Fixes * **mmap:** pre-allocate temp file before mmapping (#50) ([1ac1d647](https://github.com/zkat/cacache-rs/commit/1ac1d647e0e25e1230bca376f9f294336dd57943)) ## 11.5.2 (2023-04-01) ### Bug Fixes * **deps:** move async-attributes over to dev-dependencies ([b7c10eca](https://github.com/zkat/cacache-rs/commit/b7c10ecaaf38893507ac8cba27a8ba45eb309dce)) ## 11.5.1 (2023-04-01) ### Bug Fixes * **deps:** bump ssri too ([4950998f](https://github.com/zkat/cacache-rs/commit/4950998f77444786ef747cbacfea5214741e350d)) # `cacache` Release Changelog ## 11.5.0 (2023-04-01) This release upgrades miette and thiserror in lockstep, bringing in syn2, in order to avoid [potential issues with the upgrade](https://github.com/yaahc/eyre/pull/92). ### Features * **deps:** bump miette and thiserror ([364f9da6](https://github.com/zkat/cacache-rs/commit/364f9da65eccedc51af9e05ab326496fb98d4ed7)) ## 11.4.0 (2023-03-10) ### Features * **link:** implement hard_link_hash_sync too ([b88217cc](https://github.com/zkat/cacache-rs/commit/b88217cc548a91d0c168f8604a32b78f3366fe46)) ### Bug Fixes * **security:** bump tempfile version to 3.4.0 (#45) ([02d7f146](https://github.com/zkat/cacache-rs/commit/02d7f146f17c5db3f1c6e3970dc378493254b23d)) ## 11.3.0 (2023-03-05) ### Features * **links:** add support for hard linking from the cache (#43) ([89eee6c1](https://github.com/zkat/cacache-rs/commit/89eee6c1aebd3b271619a4b55fac7f0afc5f07bb)) ## 11.2.0 (2023-02-28) ### Features * **link_to:** Add support for linking to existing files from the cache (#41) ([ac56fd9c](https://github.com/zkat/cacache-rs/commit/ac56fd9ce6a453a707df9a0603fc041e4e785d79)) * **copy:** add support for reflink-copy and unsafe-copy (#42) ([d39e8380](https://github.com/zkat/cacache-rs/commit/d39e83801dc4f1e6479dacd50dcf1372658a598f)) ## 11.1.0 (2023-02-19) ### Features * **index:** Add support for raw index metadata and expose index functions ([9991f56b](https://github.com/zkat/cacache-rs/commit/9991f56b4c689149257dbe86357cbcab8fa228e9)) ## 11.0.2 (2023-02-19) ### Bug Fixes * **ls:** make sure `ls` sees entry deletions (#40) ([53c28952](https://github.com/zkat/cacache-rs/commit/53c289523fc0d1bc9708e3e252f84b532e405329)) ## 11.0.1 (2023-02-19) ### Features * **deps:** bump ssri to 8.0.0 ([f8cbfefc](https://github.com/zkat/cacache-rs/commit/f8cbfefc129861cece61a80dda3e5a4d9d6de053)) ## 11.0.0 (2023-01-29) This release includes a number of nice improvements to error reporting, integrating [`miette`](https://crates.io/crates/miette) to provide more helpful, contextual error messages, as well as adding [`tokio`](https://crates.io/crates/tokio) as an optional runtime, instead of `async-std`. It also includes a number of bug fixes and performance improvements. Note that this release also bumps the MSRV to 1.67.0, which will hopefully stay there for a while. Apologies if you're still on an earlier version of `rustc`: the ecosystem moved forward and ultimately, we needed to move with it. ### Bug Fixes * **write:** set tmpfile length in async writer (#35) ([6d84ff0a](https://github.com/zkat/cacache-rs/commit/6d84ff0aed53aea727b32f6b3d16ee0d045aa0e4)) * **BREAKING CHANGE**: This commit also bumps the MSRV for cacache to 1.66.1. * **deps:** only include rt-multi-threaded for tokio when running tests ([526386ad](https://github.com/zkat/cacache-rs/commit/526386ada8a6b3ebee3222edb93ef401959e7f6e)) * **msrv:** be less aggressive about MSRV bump ([dca57e11](https://github.com/zkat/cacache-rs/commit/dca57e11001f8ee06077af77dfffe3bc9df5af2a)) * **perf:** do the inner fn trick to reduce generic compilation load ([da259ae4](https://github.com/zkat/cacache-rs/commit/da259ae43233e0b0cb23a2195700d3bca2942340)) ### Features * **async:** Add tokio as an executor option (#36) ([e34dcfdc](https://github.com/zkat/cacache-rs/commit/e34dcfdc250c87010afa07279dc7d6aefce9866b)) * **errors:** integrate miette and generally improve error reporting (#38) ([c2d5390a](https://github.com/zkat/cacache-rs/commit/c2d5390a84639ddc62b0581c76dd986442cfca7d)) * **BREAKING CHANGE**: This bumps the MSRV to 1.67.0 and documents it in the README. ## 10.0.1 (2022-02-25) ### Bug Fixes * **mmap:** mmap needs equal buffer lengths (#33) ([684a3e8a](https://github.com/zkat/cacache-rs/commit/684a3e8a682d7ddc60a18671bc182373f2bc6c28)) ## 10.0.0 (2022-01-23) ### Breaking Changes * **msrv:** bump MSRV to 1.54, thanks to socket ([323ecacc](https://github.com/zkat/cacache-rs/commit/323ecacc3b1e9b70bfac5f3e0838b099e9c6f8a4)) ### Bug Fixes * **security:** replaced memmap with memmap2 (#30) ([03d4596b](https://github.com/zkat/cacache-rs/commit/03d4596baa330a057011e5238983095c08039360)) ## 9.0.0 (2021-09-16) ### Bug Fixes * **clippy:** appease the paperclip ([8d08e452](https://github.com/zkat/cacache-rs/commit/8d08e452984f0bf5a266b1939f3ce2d7ab3aed7d)) * **cleanup:** general house cleaning, bumping deps, etc ([9163a584](https://github.com/zkat/cacache-rs/commit/9163a58481e28b8ecca62268f1cf32e4e68f47db)) ### Features * **license:** change license to Apache-2.0 ([97890872](https://github.com/zkat/cacache-rs/commit/97890872d559b0c37aba8414c8f8b1055276c6cb)) * **BREAKING CHANGE**: This is a significant licensing change. Please review. ## 8.0.0 (2020-07-18) #### Breaking Changes - **write:** Use mmap for small writes (#26) ([803d0c3e](https://github.com/zkat/cacache-rs/commit/803d0c3ede199c20aec1b514daf21fab9ee68ac2), breaks [#](https://github.com/zkat/cacache-rs/issues/)). This bumps the minimum Rust version from 1.39 to 1.43 due to a dependency's breaking change in a patch version. ## 7.0.0 (2020-04-30) This release is mostly a major overhaul of the external error API, switching out of `anyhow` in favor of more bespoke error handling that works nicer in a library. #### Breaking Changes - **errors:** remove anyhow and use custom error types (#24) ([bb815f5f](https://github.com/zkat/cacache-rs/commit/bb815f5f22ea932814b8b3e120fd6cac24831d01), breaks [#](https://github.com/zkat/cacache-rs/issues/)) #### Bug Fixes - **list_sync:** make sure the public interface allows using the Item type (#25) ([88a76189](https://github.com/zkat/cacache-rs/commit/88a76189fce954949ff3026b96158f700f5e2325)) ## 6.0.0 (2019-11-12) #### Breaking Changes - **chown:** stop changing owner/group on unix platforms ([d5bb0dff](https://github.com/zkat/cacache-rs/commit/d5bb0dffb623d0a61d7680829ca36ce10ceb2f53)) - **deps:** upgrade to latest async-std and regular futures crate ([c44b781a](https://github.com/zkat/cacache-rs/commit/c44b781a34bb4f95667ccb784671060ee3c0bcca)) - **license:** upgrade to Parity 7.0 release ([b54ec598](https://github.com/zkat/cacache-rs/commit/b54ec598cb11272edd685f4db45f6ff8bbeb9747)) ## 5.0.0 (2019-10-24) #### Breaking Changes - **api:** rewrite entire API to be like std::fs (#21) ([743476b2](https://github.com/zkat/cacache-rs/commit/743476b274eb07844b7b73137770df856cd7e4c4)) - **license:** bump Parity license to 7.0.0-pre.3 ([0395b0fb](https://github.com/zkat/cacache-rs/commit/0395b0fbffc65004f2b099aee9075251c8354e06)) #### Features - **api:** rewrite entire API to be like std::fs (#21) ([743476b2](https://github.com/zkat/cacache-rs/commit/743476b274eb07844b7b73137770df856cd7e4c4)) - **license:** bump Parity license to 7.0.0-pre.3 ([0395b0fb](https://github.com/zkat/cacache-rs/commit/0395b0fbffc65004f2b099aee9075251c8354e06)) ## 4.0.0 (2019-10-21) #### Bug Fixes - **fmt:** cargo fmt --all ([38115599](https://github.com/zkat/cacache-rs/commit/38115599ca9cc9f6426b950d16399f9e03871dd3)) #### Breaking Changes - **errors:** - improved errors messaging and context (#20) ([62298cdf](https://github.com/zkat/cacache-rs/commit/62298cdf351d7ed10b54417ae7a702d07b4b4765)) - Replace failure with anyhow crate (#17) ([ee149a70](https://github.com/zkat/cacache-rs/commit/ee149a70cab9ec37951aef47a21c40a0d6efb234)) #### Features - **errors:** - improved errors messaging and context (#20) ([62298cdf](https://github.com/zkat/cacache-rs/commit/62298cdf351d7ed10b54417ae7a702d07b4b4765)) - Replace failure with anyhow crate (#17) ([ee149a70](https://github.com/zkat/cacache-rs/commit/ee149a70cab9ec37951aef47a21c40a0d6efb234)) - **license:** Add in Patron license to make proprietary stuff more clear ([fbeb6ec0](https://github.com/zkat/cacache-rs/commit/fbeb6ec0ff77e022d87dc03865d4136bbbd8fbc6)) - **rm:** Accept AsRef for keys ([64939851](https://github.com/zkat/cacache-rs/commit/649398512f339933605ed70cade3ca16962a6b26)) ## 3.0.0 (2019-10-19) #### Features - **api:** get::read -> get::data ([b02f41e0](https://github.com/zkat/cacache-rs/commit/b02f41e07fab0929006e8027395503ff001a6002)) - **async:** reorganize async APIs to be the primary APIs ([662aea9b](https://github.com/zkat/cacache-rs/commit/662aea9b5a829ca4ca9673f2d82917065d675c62)) - **get:** get::info -> get::entry ([dafc79f4](https://github.com/zkat/cacache-rs/commit/dafc79f481366f3254c13efaf101c79e018d7e19)) - **ls:** cacache::ls::all -> ls::all_sync ([c4300167](https://github.com/zkat/cacache-rs/commit/c43001674441e68dd376cf003e17167360ab670e)) #### Bug Fixes - **check:** {Async}Get::check wasn't working correctly ([d08629cf](https://github.com/zkat/cacache-rs/commit/d08629cf5547f6aad8147f319fee5d30accf89a2)) - **open:** use actual file paths instead of just cache for open APIs ([03ff1970](https://github.com/zkat/cacache-rs/commit/03ff19709ab13ff4fc61ae8b52ace93db2c9dada)) #### Breaking Changes - **api:** get::read -> get::data ([b02f41e0](https://github.com/zkat/cacache-rs/commit/b02f41e07fab0929006e8027395503ff001a6002), breaks [#](https://github.com/zkat/cacache-rs/issues/)) - **async:** reorganize async APIs to be the primary APIs ([662aea9b](https://github.com/zkat/cacache-rs/commit/662aea9b5a829ca4ca9673f2d82917065d675c62), breaks [#](https://github.com/zkat/cacache-rs/issues/)) - **get:** get::info -> get::entry ([dafc79f4](https://github.com/zkat/cacache-rs/commit/dafc79f481366f3254c13efaf101c79e018d7e19), breaks [#](https://github.com/zkat/cacache-rs/issues/)) - **ls:** cacache::ls::all -> ls::all_sync ([c4300167](https://github.com/zkat/cacache-rs/commit/c43001674441e68dd376cf003e17167360ab670e), breaks [#](https://github.com/zkat/cacache-rs/issues/)) ## 2.0.1 (2019-10-15) - Just adds some examples of the core API. ## 2.0.0 (2019-10-15) #### Features - **async:** add extra async versions of APIs (#6) ([18190bfc](https://github.com/zkat/cacache-rs/commit/18190bfc356fdf871f9f284b54fc48da32e44ead)) - **license:** - relicense to Parity+Apache ([4d9404b9](https://github.com/zkat/cacache-rs/commit/4d9404b9a606cfc52fce06999ab5a640bda8fc26)) #### Bug Fixes - **windows:** add windows support ([97f44573](https://github.com/zkat/cacache-rs/commit/97f44573d55c96172aecf4be553eba064e43d58e)) #### Breaking Changes - **license:** relicense to Parity+Apache ([4d9404b9](https://github.com/zkat/cacache-rs/commit/4d9404b9a606cfc52fce06999ab5a640bda8fc26)) ## 1.0.1 (2019-07-01) Initial History generation. #### Features - **api:** AsRef all the things! ([5af622eb](https://github.com/zkat/cacache-rs.git/commit/5af622eb30b9f177117ce2f8ad17690313fba50a)) - **content:** add baseline read functionality ([e98bfb17](https://github.com/zkat/cacache-rs.git/commit/e98bfb17da0f4b862954e5f7636ea6284cd81367)) - **error:** - Add SizeError ([0bbe080a](https://github.com/zkat/cacache-rs.git/commit/0bbe080a6ef636175ce07936ca8a7d26243509fb)) - add wrapper for atomicwrites ([dbb8c79b](https://github.com/zkat/cacache-rs.git/commit/dbb8c79b00f89e1b6303be179a6389328e1a762c)) - **errors:** add errors module ([b0464849](https://github.com/zkat/cacache-rs.git/commit/b0464849e6cd32b047bbdfaa000e961dc2d87e86)) - **exports:** re-export ssri::Algorithm and serde_json::Value ([87adc8cf](https://github.com/zkat/cacache-rs.git/commit/87adc8cf9f63211edc943e72ec28de797de574ea)) - **get:** - add get::open() and get::open_hash() ([6e9a2f9f](https://github.com/zkat/cacache-rs.git/commit/6e9a2f9f87ecfb82a7bfd90fb748053a79de4e75)) - add external cacache::get api ([d91d2141](https://github.com/zkat/cacache-rs.git/commit/d91d2141761abf0e6180dc2ecd8c486637cf9232)) - **index:** - make inserter.commit() return integrity ([257fc9b6](https://github.com/zkat/cacache-rs.git/commit/257fc9b6d0cb3f99547059821255b1719dd6be2f)) - implement delete() ([33a5dbbd](https://github.com/zkat/cacache-rs.git/commit/33a5dbbd51fc8d9ae180e8eac3f0600d8cbe37df)) - implemented find() ([44eb2acc](https://github.com/zkat/cacache-rs.git/commit/44eb2acc98b242747ff09460e0c276593dfe3840)) - implemented index::insert() ([322e68ff](https://github.com/zkat/cacache-rs.git/commit/322e68ffaa118ed519e1fe2f395b7cdfa903d91b)) - port index::insert() ([9ffc090b](https://github.com/zkat/cacache-rs.git/commit/9ffc090b3b2248def2aa9390ca1fd4028fb3663b)) - **ls:** implemented cacache::ls::all() ([b0f351ea](https://github.com/zkat/cacache-rs.git/commit/b0f351ea269778e2e0be1d1388698d7a4b97ccd0)) - **path:** ported content_path ([0f768fa5](https://github.com/zkat/cacache-rs.git/commit/0f768fa5c09445cc7dc81bcaea2639cf598f5107)) - **put:** - privatize Put and PutOpts fields ([7f1602e2](https://github.com/zkat/cacache-rs.git/commit/7f1602e28fcecc02c47a43867c43dc8b420ca120)) - make PutOpts Clone ([27ce700b](https://github.com/zkat/cacache-rs.git/commit/27ce700bd69e1b72ab761521b0ba6fe0fc93ece1)) - Add put::Put and put::PutOpts ([15f017fe](https://github.com/zkat/cacache-rs.git/commit/15f017fe2151ad70dd75fbc90bae4c1cfccc00df)) - initial implementation of cacache::put ([815d7a3c](https://github.com/zkat/cacache-rs.git/commit/815d7a3c9e880eccd89baf4565e627658c5ac553)) - **read:** - added has_content() ([bff95f20](https://github.com/zkat/cacache-rs.git/commit/bff95f20ec3f79a356a30733145f44adc99d2f83)) - added content read and read_to_string ([70cf52e1](https://github.com/zkat/cacache-rs.git/commit/70cf52e136624bbff415d2641d56331191649f17)) - **rm:** - added external rm api ([346cf5fb](https://github.com/zkat/cacache-rs.git/commit/346cf5fb2379b9486186eca6aa14b72106818fc4)) - added content/rm ([eac29d94](https://github.com/zkat/cacache-rs.git/commit/eac29d941b0e36c143d3262e891fdbf991e316d7)) - **write:** initial hack for write ([e452fdcd](https://github.com/zkat/cacache-rs.git/commit/e452fdcd16fae12d79602814979312767264a3b7)) #### Bug Fixes - **api:** use &str keys ([cf0fbe23](https://github.com/zkat/cacache-rs.git/commit/cf0fbe233f721f7ad3637eaf01207e3015f74ecd)) - **content:** make rm use our own Error ([f3b6abf4](https://github.com/zkat/cacache-rs.git/commit/f3b6abf45c0408228e3bf8a0fe1e744d0b32c0bd)) - **fmt:** - cargo fmt ([0349d115](https://github.com/zkat/cacache-rs.git/commit/0349d115f4e8d7aa59c6f7a0455b94be898efd46)) - cargo fmt ([bc56a1b3](https://github.com/zkat/cacache-rs.git/commit/bc56a1b3fee36f4ec2c3508ab34c3459904e1978)) - **index:** - get rid of last compiler warning ([22c4b301](https://github.com/zkat/cacache-rs.git/commit/22c4b3010f9a851dd53073bbe1307ecbf01ef30e)) - make fields public, too ([65040481](https://github.com/zkat/cacache-rs.git/commit/6504048181415a4818fb6f713c7f9d7be665064a)) - switch to using new error module ([6f78e00c](https://github.com/zkat/cacache-rs.git/commit/6f78e00c42d59b73c725ebb4105983aee84459ff)) - make Entry use actual Integrity objects ([7ad0633c](https://github.com/zkat/cacache-rs.git/commit/7ad0633c4363a35a53e832dcac18b4672f462cc8)) - pass references instead of using .as_path() ([fc067e95](https://github.com/zkat/cacache-rs.git/commit/fc067e95d9c8dbb29ca1732e1e6bbd7b503239cc)) - remove unneeded integrity() method ([b579be61](https://github.com/zkat/cacache-rs.git/commit/b579be617f32a26ab557fb7944da89754e40c6ea)) - **lint:** clippy told me to do this ([cba2f0d3](https://github.com/zkat/cacache-rs.git/commit/cba2f0d39afe71293742f97dcfd6c610031e5bfa)) - **put:** fix warnings ([4a6950ff](https://github.com/zkat/cacache-rs.git/commit/4a6950ff5ddf6d3f110d2cf9bedeb1ef3134d1fa)) - **write:** use shared Error type for write() ([8bf623b8](https://github.com/zkat/cacache-rs.git/commit/8bf623b8efab138f9a247edc45e477a08ab9213c)) cacache-13.0.0/CODE_OF_CONDUCT.md000064400000000000000000000234721046102023000140630ustar 00000000000000# Code of Conduct ## When Something Happens If you see a Code of Conduct violation, follow these steps: 1. Let the person know that what they did is not appropriate and ask them to stop and/or edit their message(s) or commits. 2. That person should immediately stop the behavior and correct the issue. 3. If this doesn’t happen, or if you're uncomfortable speaking up, [contact the maintainers](#contacting-maintainers). 4. As soon as available, a maintainer will look into the issue, and take [further action (see below)](#further-enforcement), starting with a warning, then temporary block, then long-term repo or organization ban. When reporting, please include any relevant details, links, screenshots, context, or other information that may be used to better understand and resolve the situation. **The maintainer team will prioritize the well-being and comfort of the recipients of the violation over the comfort of the violator.** See [some examples below](#enforcement-examples). ## Our Pledge In the interest of fostering an open and welcoming environment, we as contributors and maintainers of this project pledge to making participation in our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, technical preferences, nationality, personal appearance, race, religion, or sexual identity and orientation. ## Our Standards Examples of behavior that contributes to creating a positive environment include: * Using welcoming and inclusive language. * Being respectful of differing viewpoints and experiences. * Gracefully accepting constructive feedback. * Focusing on what is best for the community. * Showing empathy and kindness towards other community members. * Encouraging and raising up your peers in the project so you can all bask in hacks and glory. Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery and unwelcome sexual attention or advances, including when simulated online. The only exception to sexual topics is channels/spaces specifically for topics of sexual identity. * Casual mention of slavery or indentured servitude and/or false comparisons of one's occupation or situation to slavery. Please consider using or asking about alternate terminology when referring to such metaphors in technology. * Making light of/making mocking comments about trigger warnings and content warnings. * Trolling, insulting/derogatory comments, and personal or political attacks. * Public or private harassment, deliberate intimidation, or threats. * Publishing others' private information, such as a physical or electronic address, without explicit permission. This includes any sort of "outing" of any aspect of someone's identity without their consent. * Publishing private screenshots or quotes of interactions in the context of this project without all quoted users' *explicit* consent. * Publishing of private communication that doesn't have to do with reporting harrassment. * Any of the above even when [presented as "ironic" or "joking"](https://en.wikipedia.org/wiki/Hipster_racism). * Any attempt to present "reverse-ism" versions of the above as violations. Examples of reverse-isms are "reverse racism", "reverse sexism", "heterophobia", and "cisphobia". * Unsolicited explanations under the assumption that someone doesn't already know it. Ask before you teach! Don't assume what people's knowledge gaps are. * [Feigning or exaggerating surprise](https://www.recurse.com/manual#no-feigned-surprise) when someone admits to not knowing something. * "[Well-actuallies](https://www.recurse.com/manual#no-well-actuallys)" * Other conduct which could reasonably be considered inappropriate in a professional or community setting. ## Scope This Code of Conduct applies both within spaces involving this project and in other spaces involving community members. This includes the repository, its Pull Requests and Issue tracker, its Twitter community, private email communications in the context of the project, and any events where members of the project are participating, as well as adjacent communities and venues affecting the project's members. Depending on the violation, the maintainers may decide that violations of this code of conduct that have happened outside of the scope of the community may deem an individual unwelcome, and take appropriate action to maintain the comfort and safety of its members. ### Other Community Standards As a project on GitHub, this project is additionally covered by the [GitHub Community Guidelines](https://help.github.com/articles/github-community-guidelines/). Additionally, as a project hosted on npm, is is covered by [npm, Inc's Code of Conduct](https://www.npmjs.com/policies/conduct). Enforcement of those guidelines after violations overlapping with the above are the responsibility of the entities, and enforcement may happen in any or all of the services/communities. ## Maintainer Enforcement Process Once the maintainers get involved, they will follow a documented series of steps and do their best to preserve the well-being of project members. This section covers actual concrete steps. ### Contacting Maintainers You may get in touch with the maintainer team through any of the following methods: * Through email: * [kzm@sykosomatic.org](mailto:coc@zkat.tech) (Kat Marchán) * Through Twitter: * [@maybekatz](https://twitter.com/maybekatz) (Kat Marchán) ### Further Enforcement If you've already followed the [initial enforcement steps](#enforcement), these are the steps maintainers will take for further enforcement, as needed: 1. Repeat the request to stop. 2. If the person doubles down, they will have offending messages removed or edited by a maintainers given an official warning. The PR or Issue may be locked. 3. If the behavior continues or is repeated later, the person will be blocked from participating for 24 hours. 4. If the behavior continues or is repeated after the temporary block, a long-term (6-12mo) ban will be used. On top of this, maintainers may remove any offending messages, images, contributions, etc, as they deem necessary. Maintainers reserve full rights to skip any of these steps, at their discretion, if the violation is considered to be a serious and/or immediate threat to the health and well-being of members of the community. These include any threats, serious physical or verbal attacks, and other such behavior that would be completely unacceptable in any social setting that puts our members at risk. Members expelled from events or venues with any sort of paid attendance will not be refunded. ### Who Watches the Watchers? Maintainers and other leaders who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. These may include anything from removal from the maintainer team to a permanent ban from the community. Additionally, as a project hosted on both GitHub and npm, [their own Codes of Conducts may be applied against maintainers of this project](#other-community-standards), externally of this project's procedures. ### Enforcement Examples #### The Best Case The vast majority of situations work out like this. This interaction is common, and generally positive. > Alex: "Yeah I used X and it was really crazy!" > Patt (not a maintainer): "Hey, could you not use that word? What about 'ridiculous' instead?" > Alex: "oh sorry, sure." -> edits old comment to say "it was really confusing!" #### The Maintainer Case Sometimes, though, you need to get maintainers involved. Maintainers will do their best to resolve conflicts, but people who were harmed by something **will take priority**. > Patt: "Honestly, sometimes I just really hate using $library and anyone who uses it probably sucks at their job." > Alex: "Whoa there, could you dial it back a bit? There's a CoC thing about attacking folks' tech use like that." > Patt: "I'm not attacking anyone, what's your problem?" > Alex: "@maintainers hey uh. Can someone look at this issue? Patt is getting a bit aggro. I tried to nudge them about it, but nope." > KeeperOfCommitBits: (on issue) "Hey Patt, maintainer here. Could you tone it down? This sort of attack is really not okay in this space." > Patt: "Leave me alone I haven't said anything bad wtf is wrong with you." > KeeperOfCommitBits: (deletes user's comment), "@patt I mean it. Please refer to the CoC over at (URL to this CoC) if you have questions, but you can consider this an actual warning. I'd appreciate it if you reworded your messages in this thread, since they made folks there uncomfortable. Let's try and be kind, yeah?" > Patt: "@keeperofbits Okay sorry. I'm just frustrated and I'm kinda burnt out and I guess I got carried away. I'll DM Alex a note apologizing and edit my messages. Sorry for the trouble." > KeeperOfCommitBits: "@patt Thanks for that. I hear you on the stress. Burnout sucks :/. Have a good one!" #### The Nope Case > PepeTheFrog🐸: "Hi, I am a literal actual nazi and I think white supremacists are quite fashionable." > Patt: "NOOOOPE. OH NOPE NOPE." > Alex: "JFC NO. NOPE. @keeperofbits NOPE NOPE LOOK HERE" > KeeperOfCommitBits: "👀 Nope. NOPE NOPE NOPE. 🔥" > PepeTheFrog🐸 has been banned from all organization or user repositories belonging to KeeperOfCommitBits. ## Attribution This Code of Conduct was generated using [WeAllJS Code of Conduct Generator](https://npm.im/weallbehave), which is based on the [WeAllJS Code of Conduct](https://wealljs.org/code-of-conduct), which is itself based on [Contributor Covenant](http://contributor-covenant.org), version 1.4, available at [http://contributor-covenant.org/version/1/4](http://contributor-covenant.org/version/1/4), and the LGBTQ in Technology Slack [Code of Conduct](http://lgbtq.technology/coc.html). cacache-13.0.0/CONTRIBUTING.md000064400000000000000000000513051046102023000135110ustar 00000000000000# Contributing ## How do I... * [Use This Guide](#introduction)? * Ask or Say Something? 🤔🐛😱 * [Request Support](#request-support) * [Report an Error or Bug](#report-an-error-or-bug) * [Request a Feature](#request-a-feature) * Make Something? 🤓👩🏽‍💻📜🍳 * [Project Setup](#project-setup) * [Contribute Documentation](#contribute-documentation) * [Contribute Code](#contribute-code) * Manage Something ✅🙆🏼💃👔 * [Provide Support on Issues](#provide-support-on-issues) * [Label Issues](#label-issues) * [Clean Up Issues and PRs](#clean-up-issues-and-prs) * [Review Pull Requests](#review-pull-requests) * [Merge Pull Requests](#merge-pull-requests) * [Tag a Release](#tag-a-release) * [Join the Project Team](#join-the-project-team) * Add a Guide Like This One [To My Project](#attribution)? 🤖😻👻 ## Introduction Thank you so much for your interest in contributing!. All types of contributions are encouraged and valued. See the [table of contents](#toc) for different ways to help and details about how this project handles them!📝 Please make sure to read the relevant section before making your contribution! It will make it a lot easier for us maintainers to make the most of it and smooth out the experience for all involved. 💚 The [Project Team](#join-the-project-team) looks forward to your contributions. 🙌🏾✨ ## Request Support If you have a question about this project, how to use it, or just need clarification about something: * Open an Issue at https://github.com/zkat/cacache-rs/issues * Provide as much context as you can about what you're running into. * Provide project and platform versions, depending on what seems relevant. If not, please be ready to provide that information if maintainers ask for it. Once it's filed: * The project team will [label the issue](#label-issues). * Someone will try to have a response soon. * If you or the maintainers don't respond to an issue for 30 days, the [issue will be closed](#clean-up-issues-and-prs). If you want to come back to it, reply (once, please), and we'll reopen the existing issue. Please avoid filing new issues as extensions of one you already made. ## Report an Error or Bug If you run into an error or bug with the project: * Open an Issue at https://github.com/zkat/cacache-rs/issues * Include *reproduction steps* that someone else can follow to recreate the bug or error on their own. * Provide project and platform versions, depending on what seems relevant. If not, please be ready to provide that information if maintainers ask for it. Once it's filed: * The project team will [label the issue](#label-issues). * A team member will try to reproduce the issue with your provided steps. If there are no repro steps or no obvious way to reproduce the issue, the team will ask you for those steps and mark the issue as `needs-repro`. Bugs with the `needs-repro` tag will not be addressed until they are reproduced. * If the team is able to reproduce the issue, it will be marked `needs-fix`, as well as possibly other tags (such as `critical`), and the issue will be left to be [implemented by someone](#contribute-code). * If you or the maintainers don't respond to an issue for 30 days, the [issue will be closed](#clean-up-issues-and-prs). If you want to come back to it, reply (once, please), and we'll reopen the existing issue. Please avoid filing new issues as extensions of one you already made. * `critical` issues may be left open, depending on perceived immediacy and severity, even past the 30 day deadline. ## Request a Feature If the project doesn't do something you need or want it to do: * Open an Issue at https://github.com/zkat/cacache-rs/issues * Provide as much context as you can about what you're running into. * Please try and be clear about why existing features and alternatives would not work for you. Once it's filed: * The project team will [label the issue](#label-issues). * The project team will evaluate the feature request, possibly asking you more questions to understand its purpose and any relevant requirements. If the issue is closed, the team will convey their reasoning and suggest an alternative path forward. * If the feature request is accepted, it will be marked for implementation with `feature-accepted`, which can then be done by either by a core team member or by anyone in the community who wants to [contribute code](#contribute-code). Note: The team is unlikely to be able to accept every single feature request that is filed. Please understand if they need to say no. ## Project Setup So you wanna contribute some code! That's great! This project uses GitHub Pull Requests to manage contributions, so [read up on how to fork a GitHub project and file a PR](https://guides.github.com/activities/forking) if you've never done it before. If this seems like a lot or you aren't able to do all this setup, you might also be able to [edit the files directly](https://help.github.com/articles/editing-files-in-another-user-s-repository/) without having to do any of this setup. Yes, [even code](#contribute-code). If you want to go the usual route and run the project locally, though: * [Install Rust](https://www.rust-lang.org/learn/get-started) * [Fork the project](https://guides.github.com/activities/forking/#fork) Then in your terminal: * `cd path/to/your/clone` * `cargo test` And you should be ready to go! ## Contribute Documentation Documentation is a super important, critical part of this project. Docs are how we keep track of what we're doing, how, and why. It's how we stay on the same page about our policies. And it's how we tell others everything they need in order to be able to use this project -- or contribute to it. So thank you in advance. Documentation contributions of any size are welcome! Feel free to file a PR even if you're just rewording a sentence to be more clear, or fixing a spelling mistake! To contribute documentation: * [Set up the project](#project-setup). * Edit or add any relevant documentation. * Make sure your changes are formatted correctly and consistently with the rest of the documentation. * Re-read what you wrote, and run a spellchecker on it to make sure you didn't miss anything. * Write clear, concise commit message(s) using [conventional-changelog format](https://github.com/conventional-changelog/conventional-changelog-angular/blob/master/convention.md). Documentation commits should use `docs(): `. * Go to https://github.com/zkat/cacache-rs/pulls and open a new pull request with your changes. * If your PR is connected to an open issue, add a line in your PR's description that says `Fixes: #123`, where `#123` is the number of the issue you're fixing. Once you've filed the PR: * One or more maintainers will use GitHub's review feature to review your PR. * If the maintainer asks for any changes, edit your changes, push, and ask for another review. * If the maintainer decides to pass on your PR, they will thank you for the contribution and explain why they won't be accepting the changes. That's ok! We still really appreciate you taking the time to do it, and we don't take that lightly. 💚 * If your PR gets accepted, it will be marked as such, and merged into the `latest` branch soon after. Your contribution will be distributed to the masses next time the maintainers [tag a release](#tag-a-release) ## Contribute Code We like code commits a lot! They're super handy, and they keep the project going and doing the work it needs to do to be useful to others. Code contributions of just about any size are acceptable! The main difference between code contributions and documentation contributions is that contributing code requires inclusion of relevant tests for the code being added or changed. Contributions without accompanying tests will be held off until a test is added, unless the maintainers consider the specific tests to be either impossible, or way too much of a burden for such a contribution. To contribute code: * [Set up the project](#project-setup). * Make any necessary changes to the source code. * Include any [additional documentation](#contribute-documentation) the changes might need. * Write tests that verify that your contribution works as expected. * Write clear, concise commit message(s) using [conventional-changelog format](https://github.com/conventional-changelog/conventional-changelog-angular/blob/master/convention.md). * Dependency updates, additions, or removals must be in individual commits, and the message must use the format: `(deps): PKG@VERSION`, where `` is any of the usual `conventional-changelog` prefixes, at your discretion. * Go to https://github.com/zkat/cacache-rs/pulls and open a new pull request with your changes. * If your PR is connected to an open issue, add a line in your PR's description that says `Fixes: #123`, where `#123` is the number of the issue you're fixing. Once you've filed the PR: * Barring special circumstances, maintainers will not review PRs until all checks pass (Travis, AppVeyor, etc). * One or more maintainers will use GitHub's review feature to review your PR. * If the maintainer asks for any changes, edit your changes, push, and ask for another review. Additional tags (such as `needs-tests`) will be added depending on the review. * If the maintainer decides to pass on your PR, they will thank you for the contribution and explain why they won't be accepting the changes. That's ok! We still really appreciate you taking the time to do it, and we don't take that lightly. 💚 * If your PR gets accepted, it will be marked as such, and merged into the `latest` branch soon after. Your contribution will be distributed to the masses next time the maintainers [tag a release](#tag-a-release) ## Provide Support on Issues [Needs Collaborator](#join-the-project-team): none Helping out other users with their questions is a really awesome way of contributing to any community. It's not uncommon for most of the issues on an open source projects being support-related questions by users trying to understand something they ran into, or find their way around a known bug. Sometimes, the `support` label will be added to things that turn out to actually be other things, like bugs or feature requests. In that case, suss out the details with the person who filed the original issue, add a comment explaining what the bug is, and change the label from `support` to `bug` or `feature`. If you can't do this yourself, @mention a maintainer so they can do it. In order to help other folks out with their questions: * Go to the issue tracker and [filter open issues by the `support` label](https://github.com/zkat/cacache-rs/issues?q=is%3Aopen+is%3Aissue+label%3Asupport). * Read through the list until you find something that you're familiar enough with to give an answer to. * Respond to the issue with whatever details are needed to clarify the question, or get more details about what's going on. * Once the discussion wraps up and things are clarified, either close the issue, or ask the original issue filer (or a maintainer) to close it for you. Some notes on picking up support issues: * Avoid responding to issues you don't know you can answer accurately. * As much as possible, try to refer to past issues with accepted answers. Link to them from your replies with the `#123` format. * Be kind and patient with users -- often, folks who have run into confusing things might be upset or impatient. This is ok. Try to understand where they're coming from, and if you're too uncomfortable with the tone, feel free to stay away or withdraw from the issue. (note: if the user is outright hostile or is violating the CoC, [refer to the Code of Conduct](CODE_OF_CONDUCT.md) to resolve the conflict). ## Label Issues [Needs Collaborator](#join-the-project-team): Issue Tracker One of the most important tasks in handling issues is labeling them usefully and accurately. All other tasks involving issues ultimately rely on the issue being classified in such a way that relevant parties looking to do their own tasks can find them quickly and easily. In order to label issues, [open up the list of unlabeled issues](https://github.com/zkat/cacache-rs/issues?q=is%3Aopen+is%3Aissue+no%3Alabel) and, **from newest to oldest**, read through each one and apply issue labels according to the table below. If you're unsure about what label to apply, skip the issue and try the next one: don't feel obligated to label each and every issue yourself! Label | Apply When | Notes --- | --- | --- `bug` | Cases where the code (or documentation) is behaving in a way it wasn't intended to. | If something is happening that surprises the *user* but does not go against the way the code is designed, it should use the `enhancement` label. `critical` | Added to `bug` issues if the problem described makes the code completely unusable in a common situation. | `documentation` | Added to issues or pull requests that affect any of the documentation for the project. | Can be combined with other labels, such as `bug` or `enhancement`. `duplicate` | Added to issues or PRs that refer to the exact same issue as another one that's been previously labeled. | Duplicate issues should be marked and closed right away, with a message referencing the issue it's a duplicate of (with `#123`) `enhancement` | Added to [feature requests](#request-a-feature), PRs, or documentation issues that are purely additive: the code or docs currently work as expected, but a change is being requested or suggested. | `help wanted` | Applied by [Committers](#join-the-project-team) to issues and PRs that they would like to get outside help for. Generally, this means it's lower priority for the maintainer team to itself implement, but that the community is encouraged to pick up if they so desire | Never applied on first-pass labeling. `in-progress` | Applied by [Committers](#join-the-project-team) to PRs that are pending some work before they're ready for review. | The original PR submitter should @mention the team member that applied the label once the PR is complete. `performance` | This issue or PR is directly related to improving performance. | `refactor` | Added to issues or PRs that deal with cleaning up or modifying the project for the betterment of it. | `starter` | Applied by [Committers](#join-the-project-team) to issues that they consider good introductions to the project for people who have not contributed before. These are not necessarily "easy", but rather focused around how much context is necessary in order to understand what needs to be done for this project in particular. | Existing project members are expected to stay away from these unless they increase in priority. `support` | This issue is either asking a question about how to use the project, clarifying the reason for unexpected behavior, or possibly reporting a `bug` but does not have enough detail yet to determine whether it would count as such. | The label should be switched to `bug` if reliable reproduction steps are provided. Issues primarily with unintended configurations of a user's environment are not considered bugs, even if they cause crashes. `tests` | This issue or PR either requests or adds primarily tests to the project. | If a PR is pending tests, that will be handled through the [PR review process](#review-pull-requests) `wontfix` | Labelers may apply this label to issues that clearly have nothing at all to do with the project or are otherwise entirely outside of its scope/sphere of influence. [Committers](#join-the-project-team) may apply this label and close an issue or PR if they decide to pass on an otherwise relevant issue. | The issue or PR should be closed as soon as the label is applied, and a clear explanation provided of why the label was used. Contributors are free to contest the labeling, but the decision ultimately falls on committers as to whether to accept something or not. ## Clean Up Issues and PRs [Needs Collaborator](#join-the-project-team): Issue Tracker Issues and PRs can go stale after a while. Maybe they're abandoned. Maybe the team will just plain not have time to address them any time soon. In these cases, they should be closed until they're brought up again or the interaction starts over. To clean up issues and PRs: * Search the issue tracker for issues or PRs, and add the term `updated:<=YYYY-MM-DD`, where the date is 30 days before today. * Go through each issue *from oldest to newest*, and close them if **all of the following are true**: * not opened by a maintainer * not marked as `critical` * not marked as `starter` or `help wanted` (these might stick around for a while, in general, as they're intended to be available) * no explicit messages in the comments asking for it to be left open * does not belong to a milestone * Leave a message when closing saying "Cleaning up stale issue. Please reopen or ping us if and when you're ready to resume this. See https://github.com/zkat/cacache-rs/blob/latest/CONTRIBUTING.md#clean-up-issues-and-prs for more details." ## Review Pull Requests [Needs Collaborator](#join-the-project-team): Issue Tracker While anyone can comment on a PR, add feedback, etc, PRs are only *approved* by team members with Issue Tracker or higher permissions. PR reviews use [GitHub's own review feature](https://help.github.com/articles/about-pull-request-reviews/), which manages comments, approval, and review iteration. Some notes: * You may ask for minor changes ("nitpicks"), but consider whether they are really blockers to merging: try to err on the side of "approve, with comments". * *ALL PULL REQUESTS* should be covered by a test: either by a previously-failing test, an existing test that covers the entire functionality of the submitted code, or new tests to verify any new/changed behavior. All tests must also pass and follow established conventions. Test coverage should not drop, unless the specific case is considered reasonable by maintainers. * Please make sure you're familiar with the code or documentation being updated, unless it's a minor change (spellchecking, minor formatting, etc). You may @mention another project member who you think is better suited for the review, but still provide a non-approving review of your own. * Be extra kind: people who submit code/doc contributions are putting themselves in a pretty vulnerable position, and have put time and care into what they've done (even if that's not obvious to you!) -- always respond with respect, be understanding, but don't feel like you need to sacrifice your standards for their sake, either. Just don't be a jerk about it? ## Merge Pull Requests [Needs Collaborator](#join-the-project-team): Committer TBD - need to hash out a bit more of this process. ## Tag A Release [Needs Collaborator](#join-the-project-team): Committer TBD - need to hash out a bit more of this process. The most important bit here is probably that all tests must pass, and tags must use [semver](https://semver.org). ## Join the Project Team ### Ways to Join There are many ways to contribute! Most of them don't require any official status unless otherwise noted. That said, there's a couple of positions that grant special repository abilities, and this section describes how they're granted and what they do. All of the below positions are granted based on the project team's needs, as well as their consensus opinion about whether they would like to work with the person and think that they would fit well into that position. The process is relatively informal, and it's likely that people who express interest in participating can just be granted the permissions they'd like. You can spot a collaborator on the repo by looking for the `[Collaborator]` or `[Owner]` tags next to their names. Permission | Description --- | --- Issue Tracker | Granted to contributors who express a strong interest in spending time on the project's issue tracker. These tasks are mainly [labeling issues](#label-issues), [cleaning up old ones](#clean-up-issues-and-prs), and [reviewing pull requests](#review-pull-requests), as well as all the usual things non-team-member contributors can do. Issue handlers should not merge pull requests, tag releases, or directly commit code themselves: that should still be done through the usual pull request process. Becoming an Issue Handler means the project team trusts you to understand enough of the team's process and context to implement it on the issue tracker. Committer | Granted to contributors who want to handle the actual pull request merges, tagging new versions, etc. Committers should have a good level of familiarity with the codebase, and enough context to understand the implications of various changes, as well as a good sense of the will and expectations of the project team. Admin/Owner | Granted to people ultimately responsible for the project, its community, etc. ## Attribution This guide was generated using the WeAllJS `CONTRIBUTING.md` generator. [Make your own](https://npm.im/weallcontribute)! cacache-13.0.0/Cargo.toml0000644000000047050000000000100104700ustar # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO # # When uploading crates to the registry Cargo will automatically # "normalize" Cargo.toml files for maximal compatibility # with all versions of Cargo and also rewrite `path` dependencies # to registry (e.g., crates.io) dependencies. # # If you are reading this file be aware that the original Cargo.toml # will likely look very different (and much more reasonable). # See Cargo.toml.orig for the original contents. [package] edition = "2021" name = "cacache" version = "13.0.0" authors = ["Kat Marchán "] description = "Content-addressable, key-value, high-performance, on-disk cache." homepage = "https://github.com/zkat/cacache-rs" readme = "README.md" categories = [ "caching", "filesystem", ] license = "Apache-2.0" repository = "https://github.com/zkat/cacache-rs" [[bench]] name = "benchmarks" harness = false [dependencies.async-std] version = "1.10.0" features = ["unstable"] optional = true [dependencies.digest] version = "0.10.6" [dependencies.either] version = "1.6.1" [dependencies.futures] version = "0.3.17" optional = true [dependencies.hex] version = "0.4.3" [dependencies.memmap2] version = "0.5.8" optional = true [dependencies.miette] version = "5.7.0" [dependencies.reflink-copy] version = "0.1.9" [dependencies.serde] version = "1.0.130" [dependencies.serde_derive] version = "1.0.130" [dependencies.serde_json] version = "1.0.68" [dependencies.sha1] version = "0.10.5" [dependencies.sha2] version = "0.10.6" [dependencies.ssri] version = "9.0.0" [dependencies.tempfile] version = "3.4.0" [dependencies.thiserror] version = "1.0.40" [dependencies.tokio] version = "1.12.0" features = [ "fs", "io-util", "macros", "rt", ] optional = true [dependencies.tokio-stream] version = "0.1.7" features = ["io-util"] optional = true [dependencies.walkdir] version = "2.3.2" [dev-dependencies.async-attributes] version = "1.1.2" [dev-dependencies.criterion] version = "0.4.0" [dev-dependencies.lazy_static] version = "1.4.0" [dev-dependencies.tokio] version = "1.12.0" features = [ "fs", "io-util", "macros", "rt", "rt-multi-thread", ] [features] async-std = [ "dep:async-std", "futures", ] default = [ "async-std", "mmap", ] link_to = [] mmap = [ "memmap2", "libc", ] tokio-runtime = [ "tokio", "tokio-stream", "futures", ] [target."cfg(target_os = \"linux\")".dependencies.libc] version = "0.2.144" optional = true cacache-13.0.0/Cargo.toml.orig000064400000000000000000000030231046102023000141410ustar 00000000000000[package] name = "cacache" version = "13.0.0" authors = ["Kat Marchán "] edition = "2021" description = "Content-addressable, key-value, high-performance, on-disk cache." license = "Apache-2.0" repository = "https://github.com/zkat/cacache-rs" homepage = "https://github.com/zkat/cacache-rs" readme = "README.md" categories = ["caching", "filesystem"] [dependencies] async-std = { version = "1.10.0", features = ["unstable"], optional = true } digest = "0.10.6" either = "1.6.1" futures = { version = "0.3.17", optional = true } hex = "0.4.3" memmap2 = { version = "0.5.8", optional = true } miette = "5.7.0" reflink-copy = "0.1.9" serde = "1.0.130" serde_derive = "1.0.130" serde_json = "1.0.68" sha1 = "0.10.5" sha2 = "0.10.6" ssri = "9.0.0" tempfile = "3.4.0" thiserror = "1.0.40" tokio = { version = "1.12.0", features = [ "fs", "io-util", "macros", "rt", ], optional = true } tokio-stream = { version = "0.1.7", features = ["io-util"], optional = true } walkdir = "2.3.2" [target.'cfg(target_os = "linux")'.dependencies] libc = { version = "0.2.144", optional = true } [dev-dependencies] async-attributes = { version = "1.1.2" } criterion = "0.4.0" lazy_static = "1.4.0" tokio = { version = "1.12.0", features = [ "fs", "io-util", "macros", "rt", "rt-multi-thread", ] } [[bench]] name = "benchmarks" harness = false [features] default = ["async-std", "mmap"] mmap = ["memmap2", "libc"] async-std = ["dep:async-std", "futures"] link_to = [] tokio-runtime = ["tokio", "tokio-stream", "futures"] cacache-13.0.0/LICENSE.md000064400000000000000000000010621046102023000126570ustar 00000000000000Copyright 2019 Cacache Contributors Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. cacache-13.0.0/Makefile.toml000064400000000000000000000004111046102023000136620ustar 00000000000000[tasks.changelog] workspace=false install_crate="git-cliff" command = "git-cliff" args = ["--prepend", "CHANGELOG.md", "-u", "--tag", "${@}"] [tasks.release] workspace=false install_crate="cargo-release" command = "cargo" args = ["release", "--workspace", "${@}"] cacache-13.0.0/README.md000064400000000000000000000063041046102023000125360ustar 00000000000000# cacache ![CI](https://github.com/zkat/cacache-rs/workflows/CI/badge.svg) ![crates.io](https://img.shields.io/crates/v/cacache.svg) A high-performance, concurrent, content-addressable disk cache, optimized for async APIs. ## Example ```rust use cacache; use async_attributes; #[async_attributes::main] async fn main() -> Result<(), cacache::Error> { let dir = String::from("./my-cache"); // Write some data! cacache::write(&dir, "key", b"my-async-data").await?; // Get the data back! let data = cacache::read(&dir, "key").await?; assert_eq!(data, b"my-async-data"); // Clean up the data! cacache::rm::all(&dir).await?; } ``` ## Install Using [`cargo-edit`](https://crates.io/crates/cargo-edit) `$ cargo add cacache` Minimum supported Rust version is `1.43.0`. ## Documentation - [API Docs](https://docs.rs/cacache) ## Features - First-class async support, using either [`async-std`](https://crates.io/crates/async-std) or [`tokio`](https://crates.io/crates/tokio) as its runtime. Sync APIs are available but secondary. You can also use sync APIs only and remove the async runtime dependency. - `std::fs`-style API - Extraction by key or by content address (shasum, etc) - [Subresource Integrity](#integrity) web standard support - Multi-hash support - safely host sha1, sha512, etc, in a single cache - Automatic content deduplication - Atomic content writes even for large data - Fault tolerance (immune to corruption, partial writes, process races, etc) - Consistency guarantees on read and write (full data verification) - Lockless, high-concurrency cache access - Really helpful, contextual error messages - Large file support - Pretty darn fast - Arbitrary metadata storage - Cross-platform: Windows and case-(in)sensitive filesystem support - [`miette`](https://crates.io/crates/miette) integration for detailed, helpful error reporting. - Punches nazis `async-std` is the default async runtime. To use `tokio` instead, turn off default features and enable the `tokio-runtime` feature, like this: ```toml [dependencies] cacache = { version = "*", default-features = false, features = ["tokio-runtime", "mmap"] } ``` You can also remove async APIs altogether, including removing async runtime dependency: ```toml [dependencies] cacache = { version = "*", default-features = false, features = ["mmap"] } ``` Experimental support for symlinking to existing files is provided via the "link_to" feature. ## Contributing The cacache team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The [Contributor Guide](CONTRIBUTING.md) has all the information you need for everything from reporting bugs to contributing entire new features. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear. All participants and maintainers in this project are expected to follow [Code of Conduct](CODE_OF_CONDUCT.md), and just generally be excellent to each other. Happy hacking! ## MSRV The Minimum Supported Rust Version for cacache is `1.67.0`. Any changes to the MSRV will be considered breaking changes. ## License This project is licensed under [the Apache-2.0 License](LICENSE.md). cacache-13.0.0/benches/benchmarks.rs000064400000000000000000000337511046102023000153570ustar 00000000000000#[cfg(feature = "async-std")] use async_std::fs as afs; #[cfg(feature = "link_to")] use std::path::PathBuf; #[cfg(all(test, feature = "tokio"))] use tokio::fs as afs; #[cfg(all(test, feature = "async-std"))] pub use async_std::task::block_on; #[cfg(all(test, feature = "tokio"))] lazy_static::lazy_static! { static ref TOKIO_RUNTIME: tokio::runtime::Runtime = tokio::runtime::Runtime::new().unwrap(); } #[cfg(all(test, feature = "tokio"))] #[inline] pub fn block_on(future: F) -> T where F: std::future::Future, { TOKIO_RUNTIME.block_on(future) } use std::fs::{self, File}; use std::io::prelude::*; use criterion::{black_box, criterion_group, criterion_main, Criterion}; const NUM_REPEATS: usize = 10; fn baseline_read_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let path = tmp.path().join("test_file"); let data = b"hello world"; let mut fd = File::create(&path).unwrap(); fd.write_all(data).unwrap(); drop(fd); c.bench_function("baseline_read_sync", move |b| { b.iter(|| fs::read(&path).unwrap()) }); } fn baseline_read_many_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let paths: Vec<_> = (0..) .take(NUM_REPEATS) .map(|i| tmp.path().join(format!("test_file_{i}"))) .collect(); let data = b"hello world"; for path in paths.iter() { let mut fd = File::create(path).unwrap(); fd.write_all(data).unwrap(); drop(fd); } c.bench_function("baseline_read_many_sync", move |b| { b.iter(|| { for path in paths.iter() { fs::read(black_box(&path)).unwrap(); } }) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn baseline_read_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let path = tmp.path().join("test_file"); let data = b"hello world"; let mut fd = File::create(&path).unwrap(); fd.write_all(data).unwrap(); drop(fd); c.bench_function("baseline_read_async", move |b| { b.iter(|| block_on(afs::read(&path))) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn baseline_read_many_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let paths: Vec<_> = (0..) .take(NUM_REPEATS) .map(|i| tmp.path().join(format!("test_file_{i}"))) .collect(); let data = b"hello world"; for path in paths.iter() { let mut fd = File::create(path).unwrap(); fd.write_all(data).unwrap(); drop(fd); } c.bench_function("baseline_read_many_async", move |b| { b.iter(|| { let tasks = paths.iter().map(|path| afs::read(black_box(path))); block_on(futures::future::join_all(tasks)); }) }); } fn read_hash_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = b"hello world".to_vec(); let sri = cacache::write_sync(&cache, "hello", data).unwrap(); c.bench_function("get::data_hash_sync", move |b| { b.iter(|| cacache::read_hash_sync(black_box(&cache), black_box(&sri)).unwrap()) }); } fn read_hash_sync_xxh3(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = b"hello world".to_vec(); let sri = cacache::write_sync_with_algo(cacache::Algorithm::Xxh3, &cache, "hello", data).unwrap(); c.bench_function("get::data_hash_sync::xxh3", move |b| { b.iter(|| cacache::read_hash_sync(black_box(&cache), black_box(&sri)).unwrap()) }); } fn read_hash_many_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data: Vec<_> = (0..) .take(NUM_REPEATS) .map(|i| format!("test_file_{i}")) .collect(); let sris: Vec<_> = data .iter() .map(|datum| cacache::write_sync(&cache, "hello", datum).unwrap()) .collect(); c.bench_function("get::data_hash_many_sync", move |b| { b.iter(|| { for sri in sris.iter() { cacache::read_hash_sync(black_box(&cache), black_box(sri)).unwrap(); } }) }); } fn read_hash_many_sync_xxh3(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data: Vec<_> = (0..) .take(NUM_REPEATS) .map(|i| format!("test_file_{i}")) .collect(); let sris: Vec<_> = data .iter() .map(|datum| { cacache::write_sync_with_algo(cacache::Algorithm::Xxh3, &cache, "hello", datum).unwrap() }) .collect(); c.bench_function("get::data_hash_many_sync::xxh3", move |b| { b.iter(|| { for sri in sris.iter() { cacache::read_hash_sync(black_box(&cache), black_box(sri)).unwrap(); } }) }); } fn read_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = b"hello world".to_vec(); cacache::write_sync(&cache, "hello", data).unwrap(); c.bench_function("get::data_sync", move |b| { b.iter(|| cacache::read_sync(black_box(&cache), black_box(String::from("hello"))).unwrap()) }); } fn read_hash_sync_big_data(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = vec![1; 1024 * 1024 * 5]; let sri = cacache::write_sync(&cache, "hello", data).unwrap(); c.bench_function("get_hash_big_data", move |b| { b.iter(|| cacache::read_hash_sync(black_box(&cache), black_box(&sri)).unwrap()) }); } fn read_hash_sync_big_data_xxh3(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = vec![1; 1024 * 1024 * 5]; let sri = cacache::write_sync_with_algo(cacache::Algorithm::Xxh3, &cache, "hello", data).unwrap(); c.bench_function("get_hash_big_data::xxh3", move |b| { b.iter(|| cacache::read_hash_sync(black_box(&cache), black_box(&sri)).unwrap()) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn read_hash_many_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data: Vec<_> = (0..) .take(NUM_REPEATS) .map(|i| format!("test_file_{i}")) .collect(); let sris: Vec<_> = data .iter() .map(|datum| cacache::write_sync(&cache, "hello", datum).unwrap()) .collect(); c.bench_function("get::data_hash_many", move |b| { b.iter(|| { let tasks = sris .iter() .map(|sri| cacache::read_hash(black_box(&cache), black_box(sri))); block_on(futures::future::join_all(tasks)); }) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn read_hash_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = b"hello world".to_vec(); let sri = cacache::write_sync(&cache, "hello", data).unwrap(); c.bench_function("get::data_hash", move |b| { b.iter(|| block_on(cacache::read_hash(black_box(&cache), black_box(&sri))).unwrap()) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn read_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = b"hello world".to_vec(); cacache::write_sync(&cache, "hello", data).unwrap(); c.bench_function("get::data", move |b| { b.iter(|| block_on(cacache::read(black_box(&cache), black_box("hello"))).unwrap()) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn read_hash_async_big_data(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); let data = vec![1; 1024 * 1024 * 5]; let sri = cacache::write_sync(&cache, "hello", data).unwrap(); c.bench_function("get::data_big_data", move |b| { b.iter(|| block_on(cacache::read_hash(black_box(&cache), black_box(&sri))).unwrap()) }); } fn write_hash(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("put::data::sync", move |b| { b.iter_custom(|iters| { let start = std::time::Instant::now(); for i in 0..iters { cacache::write_hash_sync(&cache, format!("hello world{i}")).unwrap(); } start.elapsed() }) }); } fn write_hash_xxh3(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("put::data::sync::xxh3", move |b| { b.iter_custom(|iters| { let start = std::time::Instant::now(); for i in 0..iters { cacache::write_hash_sync_with_algo( cacache::Algorithm::Xxh3, &cache, format!("hello world{i}"), ) .unwrap(); } start.elapsed() }) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn write_hash_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("put::data", move |b| { b.iter_custom(|iters| { let start = std::time::Instant::now(); for i in 0..iters { block_on(cacache::write_hash(&cache, format!("hello world{i}"))).unwrap(); } start.elapsed() }) }); } #[cfg(any(feature = "async-std", feature = "tokio"))] fn write_hash_async_xxh3(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("put::data::xxh3", move |b| { b.iter_custom(|iters| { let start = std::time::Instant::now(); for i in 0..iters { block_on(cacache::write_hash_with_algo( cacache::Algorithm::Xxh3, &cache, format!("hello world{i}"), )) .unwrap(); } start.elapsed() }) }); } #[cfg(feature = "link_to")] fn create_tmpfile(tmp: &tempfile::TempDir, buf: &[u8]) -> PathBuf { let dir = tmp.path().to_owned(); let target = dir.join("target-file"); std::fs::create_dir_all(&target.parent().unwrap()).unwrap(); let mut file = File::create(target.clone()).unwrap(); file.write_all(buf).unwrap(); file.flush().unwrap(); target } #[cfg(feature = "link_to")] #[cfg(any(feature = "async-std", feature = "tokio"))] fn link_to_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("link_to::file", move |b| { b.iter_custom(|iters| { let start = std::time::Instant::now(); for i in 0..iters { block_on(cacache::link_to( &cache, format!("key{}", i), target.clone(), )) .unwrap(); } start.elapsed() }) }); } #[cfg(all(feature = "link_to", any(feature = "async-std", feature = "tokio")))] fn link_to_hash_async(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("link_to::file_hash", move |b| { b.iter(|| block_on(cacache::link_to_hash(&cache, target.clone())).unwrap()) }); } #[cfg(feature = "link_to")] fn link_to_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("link_to::file_sync", move |b| { b.iter_custom(|iters| { let start = std::time::Instant::now(); for i in 0..iters { cacache::link_to_sync(&cache, format!("key{}", i), target.clone()).unwrap(); } start.elapsed() }) }); } #[cfg(feature = "link_to")] fn link_to_hash_sync(c: &mut Criterion) { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let cache = tmp.path().to_owned(); c.bench_function("link_to::file_hash_sync", move |b| { b.iter(|| cacache::link_to_hash_sync(&cache, target.clone()).unwrap()) }); } criterion_group!( benches, baseline_read_sync, baseline_read_many_sync, write_hash, write_hash_xxh3, read_hash_sync, read_hash_sync_xxh3, read_hash_many_sync, read_hash_many_sync_xxh3, read_sync, read_hash_sync_big_data, read_hash_sync_big_data_xxh3, ); #[cfg(any(feature = "async-std", feature = "tokio"))] criterion_group!( benches_async, baseline_read_async, baseline_read_many_async, read_hash_async, read_hash_many_async, read_async, write_hash_async, write_hash_async_xxh3, read_hash_async_big_data, ); #[cfg(all(feature = "link_to", any(feature = "async-std", feature = "tokio")))] criterion_group!(link_to_benches_async, link_to_async, link_to_hash_async,); #[cfg(feature = "link_to")] criterion_group!(link_to_benches, link_to_sync, link_to_hash_sync); #[cfg(all( feature = "link_to", not(any(feature = "async-std", feature = "tokio")) ))] criterion_main!(benches, link_to_benches); #[cfg(all( not(feature = "link_to"), any(feature = "async-std", feature = "tokio") ))] criterion_main!(benches, benches_async); #[cfg(all(feature = "link_to", any(feature = "async-std", feature = "tokio")))] criterion_main!( benches, benches_async, link_to_benches, link_to_benches_async ); #[cfg(all( not(feature = "link_to"), not(any(feature = "async-std", feature = "tokio")) ))] criterion_main!(benches); cacache-13.0.0/cliff.toml000064400000000000000000000035571046102023000132460ustar 00000000000000# configuration file for git-cliff (0.1.0) [changelog] # changelog header header = """ # `cacache` Release Changelog """ # template for the changelog body # https://tera.netlify.app/docs/#introduction body = """ {% if version %}\ ## {{ version | replace(from="v", to="") }} ({{ timestamp | date(format="%Y-%m-%d") }}) {% else %}\ ## Unreleased {% endif %}\ {% for group, commits in commits | filter(attribute="scope") | group_by(attribute="group") %} ### {{ group | upper_first }} {% for commit in commits %} {% if commit.scope %}\ * **{{ commit.scope }}:** {{ commit.message }} ([{{ commit.id | truncate(length=8, end="") }}](https://github.com/zkat/cacache-rs/commit/{{ commit.id }})) {%- if commit.breaking %} * **BREAKING CHANGE**: {{ commit.breaking_description }} {%- endif %}\ {% endif %}\ {% endfor %} {% endfor %} """ # remove the leading and trailing whitespace from the template trim = false # changelog footer # footer = """ # # """ [git] # allow only conventional commits # https://www.conventionalcommits.org conventional_commits = true # regex for parsing and grouping commits commit_parsers = [ { message = "^feat*", group = "Features"}, { message = "^fix*", group = "Bug Fixes"}, { message = "^doc*", group = "Documentation"}, { message = "^perf*", group = "Performance"}, { message = "^refactor*", group = "Refactor"}, { message = "^style*", group = "Styling"}, { message = "^test*", group = "Testing"}, { message = "^chore\\(release\\): prepare for*", skip = true}, { message = "^chore*", group = "Miscellaneous Tasks"}, { body = ".*security", group = "Security"}, ] # filter out the commits that are not matched by commit parsers filter_commits = true # glob pattern for matching git tags # tag_pattern = "v?[0-9]*" # regex for skipping tags # skip_tags = "v0.1.0-beta.1" cacache-13.0.0/justfile000064400000000000000000000021651046102023000130300ustar 00000000000000# List available just recipes @help: just -l # Run tests on both runtimes with cargo nextest @test: echo "----------\nasync-std:\n" cargo nextest run echo "\n----------\ntokio:\n" cargo nextest run --no-default-features --features tokio-runtime # Run benchmarks with `cargo bench` @bench: echo "----------\nasync-std:\n" cargo bench echo "\n----------\ntokio:\n" cargo bench --no-default-features --features tokio-runtime # Run benchmarks with `cargo criterion` @criterion: echo "----------\nasync-std:\n" cargo criterion echo "\n----------\ntokio:\n" cargo criterion --no-default-features --features tokio-runtime # Generate a changelog with git-cliff changelog TAG: git-cliff --prepend CHANGELOG.md -u --tag {{TAG}} # Prepare a release release *args: cargo release --workspace {{args}} # Install workspace tools @install-tools: cargo install cargo-nextest cargo install cargo-release cargo install git-cliff cargo install cargo-criterion # Lint and automatically fix what we can fix @lint: cargo clippy --fix --allow-dirty --allow-staged cargo fmt cacache-13.0.0/src/async_lib.rs000064400000000000000000000065211046102023000143600ustar 00000000000000#[cfg(feature = "async-std")] pub use async_std::fs::File; #[cfg(feature = "tokio")] pub use tokio::fs::File; #[cfg(feature = "async-std")] pub use futures::io::AsyncRead; #[cfg(feature = "tokio")] pub use tokio::io::AsyncRead; #[cfg(feature = "async-std")] pub use futures::io::AsyncReadExt; #[cfg(feature = "tokio")] pub use tokio::io::AsyncReadExt; #[cfg(feature = "async-std")] pub use futures::io::AsyncBufReadExt; #[cfg(feature = "tokio")] pub use tokio::io::AsyncBufReadExt; #[cfg(feature = "async-std")] pub use futures::io::AsyncWrite; #[cfg(feature = "tokio")] pub use tokio::io::AsyncWrite; #[cfg(feature = "async-std")] pub use futures::io::AsyncWriteExt; #[cfg(feature = "tokio")] pub use tokio::io::AsyncWriteExt; #[cfg(feature = "async-std")] pub use async_std::fs::read; #[cfg(feature = "tokio")] pub use tokio::fs::read; #[cfg(feature = "async-std")] pub use async_std::fs::copy; #[cfg(feature = "tokio")] pub use tokio::fs::copy; #[cfg(feature = "async-std")] pub use async_std::fs::metadata; #[cfg(feature = "tokio")] pub use tokio::fs::metadata; #[cfg(feature = "async-std")] pub use async_std::fs::remove_file; #[cfg(feature = "tokio")] pub use tokio::fs::remove_file; #[cfg(feature = "async-std")] pub use async_std::fs::create_dir_all; #[cfg(feature = "tokio")] pub use tokio::fs::create_dir_all; #[cfg(feature = "async-std")] pub use async_std::fs::remove_dir_all; #[cfg(feature = "tokio")] pub use tokio::fs::remove_dir_all; #[cfg(feature = "async-std")] pub use async_std::fs::DirBuilder; #[cfg(feature = "tokio")] pub use tokio::fs::DirBuilder; #[cfg(feature = "async-std")] pub use async_std::fs::OpenOptions; #[cfg(feature = "tokio")] pub use tokio::fs::OpenOptions; #[cfg(feature = "async-std")] pub use async_std::io::BufReader; #[cfg(feature = "tokio")] pub use tokio::io::BufReader; #[cfg(feature = "async-std")] #[inline] pub fn lines_to_stream(lines: futures::io::Lines) -> futures::io::Lines { lines } #[cfg(feature = "tokio")] #[inline] pub fn lines_to_stream(lines: tokio::io::Lines) -> tokio_stream::wrappers::LinesStream { tokio_stream::wrappers::LinesStream::new(lines) } #[cfg(feature = "async-std")] pub use async_std::task::spawn_blocking; #[cfg(feature = "tokio")] pub use tokio::task::spawn_blocking; #[cfg(feature = "async-std")] pub use async_std::task::JoinHandle; #[cfg(feature = "async-std")] #[inline] pub fn unwrap_joinhandle_value(value: T) -> T { value } #[cfg(feature = "tokio")] pub use tokio::task::JoinHandle; #[cfg(feature = "tokio")] #[inline] pub fn unwrap_joinhandle_value(value: Result) -> T { value.unwrap() } use tempfile::NamedTempFile; use crate::errors::IoErrorExt; #[cfg(feature = "async-std")] #[inline] pub async fn create_named_tempfile(tmp_path: std::path::PathBuf) -> crate::Result { let cloned = tmp_path.clone(); spawn_blocking(|| NamedTempFile::new_in(tmp_path)) .await .with_context(|| format!("Failed to create a temp file at {}", cloned.display())) } #[cfg(feature = "tokio")] #[inline] pub async fn create_named_tempfile(tmp_path: std::path::PathBuf) -> crate::Result { let cloned = tmp_path.clone(); spawn_blocking(|| NamedTempFile::new_in(tmp_path)) .await .unwrap() .with_context(|| format!("Failed to create a temp file at {}", cloned.display())) } cacache-13.0.0/src/content/linkto.rs000064400000000000000000000210721046102023000153650ustar 00000000000000use ssri::{Algorithm, Integrity, IntegrityOpts}; use std::fs::DirBuilder; use std::fs::File; use std::path::{Path, PathBuf}; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::pin::Pin; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::task::{Context, Poll}; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncRead; use crate::content::path; use crate::errors::{IoErrorExt, Result}; #[cfg(not(any(unix, windows)))] compile_error!("Symlinking is not supported on this platform."); fn symlink_file(src: P, dst: Q) -> std::io::Result<()> where P: AsRef, Q: AsRef, { #[cfg(unix)] { use std::os::unix::fs::symlink; symlink(src, dst) } #[cfg(windows)] { use std::os::windows::fs::symlink_file; symlink_file(src, dst) } } fn create_symlink(sri: Integrity, cache: &PathBuf, target: &PathBuf) -> Result { let cpath = path::content_path(cache.as_ref(), &sri); DirBuilder::new() .recursive(true) // Safe unwrap. cpath always has multiple segments .create(cpath.parent().unwrap()) .with_context(|| { format!( "Failed to create destination directory for linked cache file, at {}", cpath.parent().unwrap().display() ) })?; if let Err(e) = symlink_file(target, &cpath) { // If symlinking fails because there's *already* a file at the desired // destination, that is ok -- all the cache should care about is that // there is **some** valid file associated with the computed integrity. if !cpath.exists() { return Err(e).with_context(|| { format!( "Failed to create cache symlink for {} at {}", target.display(), cpath.display() ) }); } } Ok(sri) } /// A `Read`-like type that calculates the integrity of a file as it is read. /// When the linker is committed, a symlink is created from the cache to the /// target file using the integrity computed from the file's contents. pub struct ToLinker { /// The path to the target file that will be symlinked from the cache. target: PathBuf, /// The path to the root of the cache directory. cache: PathBuf, /// The file descriptor to the target file. fd: File, /// The integrity builder for calculating the target file's integrity. builder: IntegrityOpts, } impl ToLinker { pub fn new(cache: &Path, algo: Algorithm, target: &Path) -> Result { let file = File::open(target) .with_context(|| format!("Failed to open reader to {}", target.display()))?; Ok(Self { target: target.to_path_buf(), cache: cache.to_path_buf(), fd: file, builder: IntegrityOpts::new().algorithm(algo), }) } /// Add the symlink to the target file from the cache. pub fn commit(self) -> Result { create_symlink(self.builder.result(), &self.cache, &self.target) } } impl std::io::Read for ToLinker { fn read(&mut self, buf: &mut [u8]) -> std::io::Result { let amt = self.fd.read(buf)?; if amt > 0 { self.builder.input(&buf[..amt]); } Ok(amt) } } /// An `AsyncRead`-like type that calculates the integrity of a file as it is /// read. When the linker is committed, a symlink is created from the cache to /// the target file using the integrity computed from the file's contents. #[cfg(any(feature = "async-std", feature = "tokio"))] pub struct AsyncToLinker { /// The path to the target file that will be symlinked from the cache. target: PathBuf, /// The path to the root of the cache directory. cache: PathBuf, /// The async-enabled file descriptor to the target file. fd: crate::async_lib::File, /// The integrity builder for calculating the target file's integrity. builder: IntegrityOpts, } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncRead for AsyncToLinker { #[cfg(feature = "async-std")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut [u8], ) -> Poll> { let amt = futures::ready!(Pin::new(&mut self.fd).poll_read(cx, buf))?; if amt > 0 { self.builder.input(&buf[..amt]); } Poll::Ready(Ok(amt)) } #[cfg(feature = "tokio")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut tokio::io::ReadBuf<'_>, ) -> Poll> { let pre_len = buf.filled().len(); futures::ready!(Pin::new(&mut self.fd).poll_read(cx, buf))?; if buf.filled().len() > pre_len { self.builder.input(&buf.filled()[pre_len..]); } Poll::Ready(Ok(())) } } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncToLinker { pub async fn new(cache: &Path, algo: Algorithm, target: &Path) -> Result { let file = crate::async_lib::File::open(target) .await .with_context(|| format!("Failed to open reader to {}", target.display()))?; Ok(Self { target: target.to_path_buf(), cache: cache.to_path_buf(), fd: file, builder: IntegrityOpts::new().algorithm(algo), }) } /// Add the symlink to the target file from the cache. pub async fn commit(self) -> Result { create_symlink(self.builder.result(), &self.cache, &self.target) } } #[cfg(test)] mod tests { use std::io::{Read, Write}; use super::*; #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; #[cfg(feature = "async-std")] use futures::io::AsyncReadExt; #[cfg(feature = "tokio")] use tokio::io::AsyncReadExt; fn create_tmpfile(tmp: &tempfile::TempDir, buf: &[u8]) -> PathBuf { let dir = tmp.path().to_owned(); let target = dir.join("target-file"); std::fs::create_dir_all(&target.parent().unwrap()).unwrap(); let mut file = File::create(&target).unwrap(); file.write_all(buf).unwrap(); file.flush().unwrap(); target } #[test] fn basic_link() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut linker = ToLinker::new(&dir, Algorithm::Sha256, &target).unwrap(); // read all of the data from the linker, which will calculate the integrity // hash. let mut buf = Vec::new(); linker.read_to_end(&mut buf).unwrap(); assert_eq!(buf, b"hello world"); // commit the linker, creating a symlink in the cache and an integrity // hash. let sri = linker.commit().unwrap(); assert_eq!(sri.to_string(), Integrity::from(b"hello world").to_string()); let cpath = path::content_path(&dir, &sri); assert!(cpath.exists()); let metadata = std::fs::symlink_metadata(&cpath).unwrap(); let file_type = metadata.file_type(); assert!(file_type.is_symlink()); assert_eq!(std::fs::read(cpath).unwrap(), b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn basic_async_link() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut linker = AsyncToLinker::new(&dir, Algorithm::Sha256, &target) .await .unwrap(); // read all of the data from the linker, which will calculate the integrity // hash. let mut buf: Vec = Vec::new(); AsyncReadExt::read_to_end(&mut linker, &mut buf) .await .unwrap(); assert_eq!(buf, b"hello world"); // commit the linker, creating a symlink in the cache and an integrity // hash. let sri = linker.commit().await.unwrap(); assert_eq!(sri.to_string(), Integrity::from(b"hello world").to_string()); let cpath = path::content_path(&dir, &sri); assert!(cpath.exists()); let metadata = std::fs::symlink_metadata(&cpath).unwrap(); let file_type = metadata.file_type(); assert!(file_type.is_symlink()); assert_eq!(std::fs::read(cpath).unwrap(), b"hello world"); } } cacache-13.0.0/src/content/mod.rs000064400000000000000000000001441046102023000146410ustar 00000000000000pub mod path; pub mod read; pub mod rm; pub mod write; #[cfg(feature = "link_to")] pub mod linkto; cacache-13.0.0/src/content/path.rs000064400000000000000000000023111046102023000150140ustar 00000000000000use ssri::Integrity; use std::path::{Path, PathBuf}; const CONTENT_VERSION: &str = "2"; // Current format of content file path: // // sha512-BaSE64Hex= -> // ~/.my-cache/content-v2/sha512/ba/da/55deadbeefc0ffee // pub fn content_path(cache: &Path, sri: &Integrity) -> PathBuf { let mut path = PathBuf::new(); let (algo, hex) = sri.to_hex(); path.push(cache); path.push(format!("content-v{CONTENT_VERSION}")); path.push(algo.to_string()); path.push(&hex[0..2]); path.push(&hex[2..4]); path.push(&hex[4..]); path } #[cfg(test)] mod tests { use super::*; use ssri::Integrity; use std::path::Path; #[test] fn basic_test() { let sri = Integrity::from(b"hello world"); let cpath = content_path(Path::new("~/.my-cache"), &sri); let mut wanted = PathBuf::new(); wanted.push("~/.my-cache"); wanted.push(format!("content-v{CONTENT_VERSION}")); wanted.push("sha256"); wanted.push("b9"); wanted.push("4d"); wanted.push("27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"); assert_eq!(cpath.to_str().unwrap(), wanted.to_str().unwrap()); } } cacache-13.0.0/src/content/read.rs000064400000000000000000000227401046102023000150030ustar 00000000000000use std::fs::{self, File}; use std::io::Read; use std::path::Path; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::pin::Pin; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::task::{Context, Poll}; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncReadExt; use ssri::{Algorithm, Integrity, IntegrityChecker}; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncRead; use crate::content::path; use crate::errors::{IoErrorExt, Result}; pub struct Reader { fd: File, checker: IntegrityChecker, } impl std::io::Read for Reader { fn read(&mut self, buf: &mut [u8]) -> std::io::Result { let amt = self.fd.read(buf)?; self.checker.input(&buf[..amt]); Ok(amt) } } impl Reader { pub fn check(self) -> Result { Ok(self.checker.result()?) } } #[cfg(any(feature = "async-std", feature = "tokio"))] pub struct AsyncReader { fd: crate::async_lib::File, checker: IntegrityChecker, } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncRead for AsyncReader { #[cfg(feature = "async-std")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut [u8], ) -> Poll> { let amt = futures::ready!(Pin::new(&mut self.fd).poll_read(cx, buf))?; self.checker.input(&buf[..amt]); Poll::Ready(Ok(amt)) } #[cfg(feature = "tokio")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut tokio::io::ReadBuf<'_>, ) -> Poll> { let pre_len = buf.filled().len(); futures::ready!(Pin::new(&mut self.fd).poll_read(cx, buf))?; let post_len = buf.filled().len(); if post_len - pre_len == 0 { return Poll::Ready(Ok(())); } self.checker.input(&buf.filled()[pre_len..]); Poll::Ready(Ok(())) } } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncReader { pub fn check(self) -> Result { Ok(self.checker.result()?) } } pub fn open(cache: &Path, sri: Integrity) -> Result { let cpath = path::content_path(cache, &sri); Ok(Reader { fd: File::open(cpath).with_context(|| { format!( "Failed to open reader to {}", path::content_path(cache, &sri).display() ) })?, checker: IntegrityChecker::new(sri), }) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn open_async(cache: &Path, sri: Integrity) -> Result { let cpath = path::content_path(cache, &sri); Ok(AsyncReader { fd: crate::async_lib::File::open(cpath).await.with_context(|| { format!( "Failed to open reader to {}", path::content_path(cache, &sri).display() ) })?, checker: IntegrityChecker::new(sri), }) } pub fn read(cache: &Path, sri: &Integrity) -> Result> { let cpath = path::content_path(cache, sri); let ret = fs::read(cpath).with_context(|| { format!( "Failed to read contents for file at {}", path::content_path(cache, sri).display() ) })?; sri.check(&ret)?; Ok(ret) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn read_async<'a>(cache: &'a Path, sri: &'a Integrity) -> Result> { let cpath = path::content_path(cache, sri); let ret = crate::async_lib::read(&cpath).await.with_context(|| { format!( "Failed to read contents for file at {}", path::content_path(cache, sri).display() ) })?; sri.check(&ret)?; Ok(ret) } pub fn reflink_unchecked(cache: &Path, sri: &Integrity, to: &Path) -> Result<()> { let cpath = path::content_path(cache, sri); reflink_copy::reflink(cpath, to).with_context(|| { format!( "Failed to reflink cache contents from {} to {}", path::content_path(cache, sri).display(), to.display() ) })?; Ok(()) } pub fn reflink(cache: &Path, sri: &Integrity, to: &Path) -> Result<()> { let mut reader = open(cache, sri.clone())?; let mut buf: [u8; 1024] = [0; 1024]; loop { let read = reader.read(&mut buf).with_context(|| { format!( "Failed to read cache contents while verifying integrity for {}", path::content_path(cache, sri).display() ) })?; if read == 0 { break; } } reader.check()?; reflink_unchecked(cache, sri, to) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn reflink_async(cache: &Path, sri: &Integrity, to: &Path) -> Result<()> { let mut reader = open_async(cache, sri.clone()).await?; let mut buf = [0u8; 1024 * 8]; loop { let read = AsyncReadExt::read(&mut reader, &mut buf) .await .with_context(|| { format!( "Failed to read cache contents while verifying integrity for {}", path::content_path(cache, sri).display() ) })?; if read == 0 { break; } } reader.check()?; reflink_unchecked(cache, sri, to) } pub fn copy_unchecked(cache: &Path, sri: &Integrity, to: &Path) -> Result { let cpath = path::content_path(cache, sri); std::fs::copy(cpath, to).with_context(|| { format!( "Failed to copy cache contents from {} to {}", path::content_path(cache, sri).display(), to.display() ) }) } pub fn copy(cache: &Path, sri: &Integrity, to: &Path) -> Result { let mut reader = open(cache, sri.clone())?; let mut buf: [u8; 1024] = [0; 1024]; let mut size = 0; loop { let read = reader.read(&mut buf).with_context(|| { format!( "Failed to read cache contents while verifying integrity for {}", path::content_path(cache, sri).display() ) })?; size += read; if read == 0 { break; } } reader.check()?; copy_unchecked(cache, sri, to)?; Ok(size as u64) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn copy_unchecked_async<'a>( cache: &'a Path, sri: &'a Integrity, to: &'a Path, ) -> Result { let cpath = path::content_path(cache, sri); crate::async_lib::copy(&cpath, to).await.with_context(|| { format!( "Failed to copy cache contents from {} to {}", path::content_path(cache, sri).display(), to.display() ) }) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn copy_async<'a>(cache: &'a Path, sri: &'a Integrity, to: &'a Path) -> Result { let mut reader = open_async(cache, sri.clone()).await?; let mut buf: [u8; 1024] = [0; 1024]; let mut size = 0; loop { let read = AsyncReadExt::read(&mut reader, &mut buf) .await .with_context(|| { format!( "Failed to read cache contents while verifying integrity for {}", path::content_path(cache, sri).display() ) })?; size += read; if read == 0 { break; } } reader.check()?; copy_unchecked_async(cache, sri, to).await?; Ok(size as u64) } pub fn hard_link_unchecked(cache: &Path, sri: &Integrity, to: &Path) -> Result<()> { let cpath = path::content_path(cache, sri); std::fs::hard_link(cpath, to).with_context(|| { format!( "Failed to link cache contents from {} to {}", path::content_path(cache, sri).display(), to.display() ) })?; Ok(()) } pub fn hard_link(cache: &Path, sri: &Integrity, to: &Path) -> Result<()> { hard_link_unchecked(cache, sri, to)?; let mut reader = open(cache, sri.clone())?; let mut buf = [0u8; 1024 * 8]; loop { let read = reader.read(&mut buf).with_context(|| { format!( "Failed to read cache contents while verifying integrity for {}", path::content_path(cache, sri).display() ) })?; if read == 0 { break; } } reader.check()?; Ok(()) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn hard_link_async(cache: &Path, sri: &Integrity, to: &Path) -> Result<()> { let mut reader = open_async(cache, sri.clone()).await?; let mut buf = [0u8; 1024 * 8]; loop { let read = AsyncReadExt::read(&mut reader, &mut buf) .await .with_context(|| { format!( "Failed to read cache contents while verifying integrity for {}", path::content_path(cache, sri).display() ) })?; if read == 0 { break; } } reader.check()?; hard_link_unchecked(cache, sri, to)?; Ok(()) } pub fn has_content(cache: &Path, sri: &Integrity) -> Option { if path::content_path(cache, sri).exists() { Some(sri.clone()) } else { None } } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn has_content_async(cache: &Path, sri: &Integrity) -> Option { if crate::async_lib::metadata(path::content_path(cache, sri)) .await .is_ok() { Some(sri.clone()) } else { None } } cacache-13.0.0/src/content/rm.rs000064400000000000000000000014401046102023000145000ustar 00000000000000use std::fs; use std::path::Path; use ssri::Integrity; use crate::content::path; use crate::errors::{IoErrorExt, Result}; pub fn rm(cache: &Path, sri: &Integrity) -> Result<()> { fs::remove_file(path::content_path(cache, sri)).with_context(|| { format!( "Failed to remove cache file {}", path::content_path(cache, sri).display() ) })?; Ok(()) } #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn rm_async(cache: &Path, sri: &Integrity) -> Result<()> { crate::async_lib::remove_file(path::content_path(cache, sri)) .await .with_context(|| { format!( "Failed to remove cache file {}", path::content_path(cache, sri).display() ) })?; Ok(()) } cacache-13.0.0/src/content/write.rs000064400000000000000000000451051046102023000152220ustar 00000000000000use std::fs::DirBuilder; use std::io::prelude::*; use std::path::{Path, PathBuf}; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::pin::Pin; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::sync::Mutex; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::task::{Context, Poll}; #[cfg(any(feature = "async-std", feature = "tokio"))] use futures::prelude::*; #[cfg(feature = "mmap")] use memmap2::MmapMut; use ssri::{Algorithm, Integrity, IntegrityOpts}; use tempfile::NamedTempFile; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::{AsyncWrite, JoinHandle}; use crate::content::path; use crate::errors::{IoErrorExt, Result}; #[cfg(feature = "mmap")] pub const MAX_MMAP_SIZE: usize = 1024 * 1024; #[cfg(not(feature = "mmap"))] struct MmapMut; #[cfg(not(feature = "mmap"))] impl MmapMut { #[allow(dead_code)] fn flush_async(&self) -> std::io::Result<()> { panic!() } fn copy_from_slice(&self, _: &[u8]) { panic!() } } pub struct Writer { cache: PathBuf, builder: IntegrityOpts, mmap: Option, tmpfile: NamedTempFile, } impl Writer { pub fn new(cache: &Path, algo: Algorithm, size: Option) -> Result { let cache_path = cache.to_path_buf(); let mut tmp_path = cache_path.clone(); tmp_path.push("tmp"); DirBuilder::new() .recursive(true) .create(&tmp_path) .with_context(|| { format!( "Failed to create cache directory for temporary files, at {}", tmp_path.display() ) })?; let tmp_path_clone = tmp_path.clone(); let mut tmpfile = NamedTempFile::new_in(tmp_path).with_context(|| { format!( "Failed to create temp file while initializing a writer, inside {}", tmp_path_clone.display() ) })?; let mmap = make_mmap(&mut tmpfile, size)?; Ok(Writer { cache: cache_path, builder: IntegrityOpts::new().algorithm(algo), tmpfile, mmap, }) } pub fn close(self) -> Result { let sri = self.builder.result(); let cpath = path::content_path(&self.cache, &sri); DirBuilder::new() .recursive(true) // Safe unwrap. cpath always has multiple segments .create(cpath.parent().unwrap()) .with_context(|| { format!( "Failed to create destination directory for cache contents, at {}", path::content_path(&self.cache, &sri) .parent() .unwrap() .display() ) })?; let res = self.tmpfile.persist(&cpath); match res { Ok(_) => {} Err(e) => { // We might run into conflicts sometimes when persisting files. // This is ok. We can deal. Let's just make sure the destination // file actually exists, and we can move on. if !cpath.exists() { return Err(e.error).with_context(|| { format!( "Failed to persist cache contents while closing writer, at {}", path::content_path(&self.cache, &sri).display() ) })?; } } } Ok(sri) } } impl Write for Writer { fn write(&mut self, buf: &[u8]) -> std::io::Result { self.builder.input(buf); if let Some(mmap) = &mut self.mmap { mmap.copy_from_slice(buf); Ok(buf.len()) } else { self.tmpfile.write(buf) } } fn flush(&mut self) -> std::io::Result<()> { self.tmpfile.flush() } } #[cfg(any(feature = "async-std", feature = "tokio"))] pub struct AsyncWriter(Mutex); #[cfg(any(feature = "async-std", feature = "tokio"))] enum State { Idle(Option), Busy(JoinHandle), } #[cfg(any(feature = "async-std", feature = "tokio"))] struct Inner { cache: PathBuf, builder: IntegrityOpts, tmpfile: NamedTempFile, mmap: Option, buf: Vec, last_op: Option, } #[cfg(any(feature = "async-std", feature = "tokio"))] enum Operation { Write(std::io::Result), Flush(std::io::Result<()>), } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncWriter { #[allow(clippy::new_ret_no_self)] #[allow(clippy::needless_lifetimes)] pub async fn new(cache: &Path, algo: Algorithm, size: Option) -> Result { let cache_path = cache.to_path_buf(); let mut tmp_path = cache_path.clone(); tmp_path.push("tmp"); crate::async_lib::DirBuilder::new() .recursive(true) .create(&tmp_path) .await .with_context(|| { format!( "Failed to create cache directory for temporary files, at {}", tmp_path.display() ) })?; let mut tmpfile = crate::async_lib::create_named_tempfile(tmp_path).await?; let mmap = make_mmap(&mut tmpfile, size)?; Ok(AsyncWriter(Mutex::new(State::Idle(Some(Inner { cache: cache_path, builder: IntegrityOpts::new().algorithm(algo), mmap, tmpfile, buf: vec![], last_op: None, }))))) } pub async fn close(self) -> Result { // NOTE: How do I even get access to `inner` safely??? // let inner = ???; // Blocking, but should be a very fast op. futures::future::poll_fn(|cx| { let state = &mut *self.0.lock().unwrap(); loop { match state { State::Idle(opt) => match opt.take() { None => return Poll::Ready(None), Some(inner) => { let (s, r) = futures::channel::oneshot::channel(); let tmpfile = inner.tmpfile; let sri = inner.builder.result(); let cpath = path::content_path(&inner.cache, &sri); // Start the operation asynchronously. *state = State::Busy(crate::async_lib::spawn_blocking(|| { let res = std::fs::DirBuilder::new() .recursive(true) // Safe unwrap. cpath always has multiple segments .create(cpath.parent().unwrap()) .with_context(|| { format!( "building directory {} failed", cpath.parent().unwrap().display() ) }); if res.is_err() { let _ = s.send(res.map(|_| sri)); } else { let res = tmpfile .persist(&cpath) .map_err(|e| e.error) .with_context(|| { format!("persisting file {} failed", cpath.display()) }); if res.is_err() { // We might run into conflicts // sometimes when persisting files. // This is ok. We can deal. Let's just // make sure the destination file // actually exists, and we can move // on. let _ = s.send( std::fs::metadata(cpath) .with_context(|| { String::from("File still doesn't exist") }) .map(|_| sri), ); } else { let _ = s.send(res.map(|_| sri)); } } State::Idle(None) })); return Poll::Ready(Some(r)); } }, // Poll the asynchronous operation the file is currently blocked on. State::Busy(task) => { *state = crate::async_lib::unwrap_joinhandle_value(futures::ready!( Pin::new(task).poll(cx) )) } } } }) .map(|opt| opt.ok_or_else(|| crate::errors::io_error("file closed"))) .await .with_context(|| "Error while closing cache contents".to_string())? .await .map_err(|_| crate::errors::io_error("Operation cancelled")) .with_context(|| "Error while closing cache contents".to_string())? } } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncWrite for AsyncWriter { fn poll_write( self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &[u8], ) -> Poll> { let state = &mut *self.0.lock().unwrap(); loop { match state { State::Idle(opt) => { // Grab a reference to the inner representation of the file or return an error // if the file is closed. let inner = opt .as_mut() .ok_or_else(|| crate::errors::io_error("file closed"))?; // Check if the operation has completed. if let Some(Operation::Write(res)) = inner.last_op.take() { let n = res?; // If more data was written than is available in the buffer, let's retry // the write operation. if n <= buf.len() { return Poll::Ready(Ok(n)); } } else { let mut inner = opt.take().unwrap(); // Set the length of the inner buffer to the length of the provided buffer. if inner.buf.len() < buf.len() { inner.buf.reserve(buf.len() - inner.buf.len()); } unsafe { inner.buf.set_len(buf.len()); } // Copy the data to write into the inner buffer. inner.buf[..buf.len()].copy_from_slice(buf); // Start the operation asynchronously. *state = State::Busy(crate::async_lib::spawn_blocking(|| { inner.builder.input(&inner.buf); if let Some(mmap) = &mut inner.mmap { mmap.copy_from_slice(&inner.buf); inner.last_op = Some(Operation::Write(Ok(inner.buf.len()))); State::Idle(Some(inner)) } else { let res = inner.tmpfile.write(&inner.buf); inner.last_op = Some(Operation::Write(res)); State::Idle(Some(inner)) } })); } } // Poll the asynchronous operation the file is currently blocked on. State::Busy(task) => { *state = crate::async_lib::unwrap_joinhandle_value(futures::ready!(Pin::new( task ) .poll(cx))) } } } } fn poll_flush(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { let state = &mut *self.0.lock().unwrap(); loop { match state { State::Idle(opt) => { // Grab a reference to the inner representation of the file or return if the // file is closed. let inner = match opt.as_mut() { None => return Poll::Ready(Ok(())), Some(s) => s, }; // Check if the operation has completed. if let Some(Operation::Flush(res)) = inner.last_op.take() { return Poll::Ready(res); } else { let mut inner = opt.take().unwrap(); if let Some(mmap) = &inner.mmap { match mmap.flush_async() { Ok(_) => (), Err(e) => return Poll::Ready(Err(e)), }; } // Start the operation asynchronously. *state = State::Busy(crate::async_lib::spawn_blocking(|| { let res = inner.tmpfile.flush(); inner.last_op = Some(Operation::Flush(res)); State::Idle(Some(inner)) })); } } // Poll the asynchronous operation the file is currently blocked on. State::Busy(task) => { *state = crate::async_lib::unwrap_joinhandle_value(futures::ready!(Pin::new( task ) .poll(cx))) } } } } #[cfg(feature = "async-std")] fn poll_close(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { self.poll_close_impl(cx) } #[cfg(feature = "tokio")] fn poll_shutdown(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { self.poll_close_impl(cx) } } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncWriter { #[inline] fn poll_close_impl( self: Pin<&mut Self>, cx: &mut std::task::Context<'_>, ) -> Poll> { let state = &mut *self.0.lock().unwrap(); loop { match state { State::Idle(opt) => { // Grab a reference to the inner representation of the file or return if the // file is closed. let inner = match opt.take() { None => return Poll::Ready(Ok(())), Some(s) => s, }; // Start the operation asynchronously. *state = State::Busy(crate::async_lib::spawn_blocking(|| { drop(inner); State::Idle(None) })); } // Poll the asynchronous operation the file is currently blocked on. State::Busy(task) => { *state = crate::async_lib::unwrap_joinhandle_value(futures::ready!(Pin::new( task ) .poll(cx))) } } } } } #[cfg(feature = "mmap")] fn make_mmap(tmpfile: &mut NamedTempFile, size: Option) -> Result> { if let Some(size @ 0..=MAX_MMAP_SIZE) = size { allocate_file(tmpfile.as_file(), size).with_context(|| { format!( "Failed to configure file length for temp file at {}", tmpfile.path().display() ) })?; Ok(unsafe { MmapMut::map_mut(tmpfile.as_file()).ok() }) } else { Ok(None) } } #[cfg(feature = "mmap")] #[cfg(target_os = "linux")] fn allocate_file(file: &std::fs::File, size: usize) -> std::io::Result<()> { use std::io::{Error, ErrorKind}; use std::os::fd::AsRawFd; let fd = file.as_raw_fd(); match unsafe { libc::posix_fallocate64(fd, 0, size as i64) } { 0 => Ok(()), libc::ENOSPC => Err(Error::new( ErrorKind::Other, // ErrorKind::StorageFull is unstable "cannot allocate file: no space left on device", )), err => Err(Error::new( ErrorKind::Other, format!("posix_fallocate64 failed with code {err}"), )), } } #[cfg(feature = "mmap")] #[cfg(not(target_os = "linux"))] fn allocate_file(file: &std::fs::File, size: usize) -> std::io::Result<()> { file.set_len(size as u64) } #[cfg(not(feature = "mmap"))] fn make_mmap(_: &mut NamedTempFile, _: Option) -> Result> { Ok(None) } #[cfg(test)] mod tests { use super::*; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncWriteExt; use tempfile; #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; #[test] fn basic_write() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut writer = Writer::new(&dir, Algorithm::Sha256, None).unwrap(); writer.write_all(b"hello world").unwrap(); let sri = writer.close().unwrap(); assert_eq!(sri.to_string(), Integrity::from(b"hello world").to_string()); assert_eq!( std::fs::read(path::content_path(&dir, &sri)).unwrap(), b"hello world" ); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn basic_async_write() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut writer = AsyncWriter::new(&dir, Algorithm::Sha256, None) .await .unwrap(); writer.write_all(b"hello world").await.unwrap(); let sri = writer.close().await.unwrap(); assert_eq!(sri.to_string(), Integrity::from(b"hello world").to_string()); assert_eq!( std::fs::read(path::content_path(&dir, &sri)).unwrap(), b"hello world" ); } } cacache-13.0.0/src/errors.rs000064400000000000000000000037741046102023000137400ustar 00000000000000use std::path::PathBuf; use miette::Diagnostic; use thiserror::Error; /// Error type returned by all API calls. #[derive(Error, Debug, Diagnostic)] pub enum Error { /// Returned when an index entry could not be found during /// lookup. #[error("Entry not found for key {1:?} in cache {0:?}")] #[diagnostic(code(cacache::entry_not_found), url(docsrs))] EntryNotFound(PathBuf, String), /// Returned when a size check has failed. #[error("Size check failed.\n\tWanted: {0}\n\tActual: {1}")] #[diagnostic(code(cacache::size_mismatch), url(docsrs))] SizeMismatch(usize, usize), /// Returned when a general IO error has occurred. #[error("{1}")] #[diagnostic(code(cacache::io_error), url(docsrs))] IoError(#[source] std::io::Error, String), /// Returned when a general serde error has occurred. #[error("{1}")] #[diagnostic(code(cacache::serde_error), url(docsrs))] SerdeError(#[source] serde_json::Error, String), /// Returned when an integrity check has failed. #[error(transparent)] #[diagnostic(code(cacache::integrity_error), url(docsrs))] IntegrityError(#[from] ssri::Error), } /// The result type returned by calls to this library pub type Result = std::result::Result; pub trait IoErrorExt { fn with_context String>(self, f: F) -> Result; } impl IoErrorExt for std::result::Result { fn with_context String>(self, f: F) -> Result { match self { Ok(t) => Ok(t), Err(e) => Err(Error::IoError(e, f())), } } } impl IoErrorExt for std::result::Result { fn with_context String>(self, f: F) -> Result { match self { Ok(t) => Ok(t), Err(e) => Err(Error::SerdeError(e, f())), } } } pub fn io_error(err: impl Into>) -> std::io::Error { std::io::Error::new(std::io::ErrorKind::Other, err) } cacache-13.0.0/src/get.rs000064400000000000000000001010771046102023000131760ustar 00000000000000//! Functions for reading from cache. use std::path::Path; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::pin::Pin; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::task::{Context as TaskContext, Poll}; use ssri::{Algorithm, Integrity}; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncRead; use crate::content::read; use crate::errors::{Error, Result}; use crate::index::{self, Metadata}; // --------- // Async API // --------- /// File handle for reading data asynchronously. /// /// Make sure to call `.check()` when done reading to verify that the /// extracted data passes integrity verification. #[cfg(any(feature = "async-std", feature = "tokio"))] pub struct Reader { reader: read::AsyncReader, } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncRead for Reader { #[cfg(feature = "async-std")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>, buf: &mut [u8], ) -> Poll> { Pin::new(&mut self.reader).poll_read(cx, buf) } #[cfg(feature = "tokio")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>, buf: &mut tokio::io::ReadBuf<'_>, ) -> Poll> { Pin::new(&mut self.reader).poll_read(cx, buf) } } #[cfg(any(feature = "async-std", feature = "tokio"))] impl Reader { /// Checks that data read from disk passes integrity checks. Returns the /// algorithm that was used verified the data. Should be called only after /// all data has been read from disk. /// /// This check is very cheap, since most of the verification is done on /// the fly. This simply finalizes verification, and is always /// synchronous. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let mut fd = cacache::Reader::open("./my-cache", "my-key").await?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).await.expect("Failed to read to string"); /// // Remember to check that the data you got was correct! /// fd.check()?; /// Ok(()) /// } /// ``` pub fn check(self) -> Result { self.reader.check() } /// Opens a new file handle into the cache, looking it up in the index using /// `key`. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let mut fd = cacache::Reader::open("./my-cache", "my-key").await?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).await.expect("Failed to read to string"); /// // Remember to check that the data you got was correct! /// fd.check()?; /// Ok(()) /// } /// ``` pub async fn open(cache: P, key: K) -> Result where P: AsRef, K: AsRef, { async fn inner(cache: &Path, key: &str) -> Result { if let Some(entry) = index::find_async(cache, key).await? { Reader::open_hash(cache, entry.integrity).await } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref()).await } /// Opens a new file handle into the cache, based on its integrity address. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "key", b"hello world").await?; /// let mut fd = cacache::Reader::open_hash("./my-cache", sri).await?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).await.expect("Failed to read to string"); /// // Remember to check that the data you got was correct! /// fd.check()?; /// Ok(()) /// } /// ``` pub async fn open_hash

(cache: P, sri: Integrity) -> Result where P: AsRef, { Ok(Reader { reader: read::open_async(cache.as_ref(), sri).await?, }) } } /// Reads the entire contents of a cache file into a bytes vector, looking the /// data up by key. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let data: Vec = cacache::read("./my-cache", "my-key").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn read(cache: P, key: K) -> Result> where P: AsRef, K: AsRef, { async fn inner(cache: &Path, key: &str) -> Result> { if let Some(entry) = index::find_async(cache, key).await? { read_hash(cache, &entry.integrity).await } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref()).await } /// Reads the entire contents of a cache file into a bytes vector, looking the /// data up by its content address. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello").await?; /// let data: Vec = cacache::read_hash("./my-cache", &sri).await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn read_hash

(cache: P, sri: &Integrity) -> Result> where P: AsRef, { read::read_async(cache.as_ref(), sri).await } /// Copies cache data to a specified location. Returns the number of bytes /// copied. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::copy("./my-cache", "my-key", "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn copy(cache: P, key: K, to: Q) -> Result where P: AsRef, K: AsRef, Q: AsRef, { async fn inner(cache: &Path, key: &str, to: &Path) -> Result { if let Some(entry) = index::find_async(cache, key).await? { copy_hash(cache, &entry.integrity, to).await } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()).await } /// Copies cache data to a specified location. Cache data will not be checked /// during copy. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::copy_unchecked("./my-cache", "my-key", "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn copy_unchecked(cache: P, key: K, to: Q) -> Result where P: AsRef, K: AsRef, Q: AsRef, { async fn inner(cache: &Path, key: &str, to: &Path) -> Result { if let Some(entry) = index::find_async(cache, key).await? { copy_hash_unchecked(cache, &entry.integrity, to).await } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()).await } /// Copies a cache data by hash to a specified location. Returns the number of /// bytes copied. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello world").await?; /// cacache::copy_hash("./my-cache", &sri, "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn copy_hash(cache: P, sri: &Integrity, to: Q) -> Result where P: AsRef, Q: AsRef, { read::copy_async(cache.as_ref(), sri, to.as_ref()).await } /// Copies a cache data by hash to a specified location. Copied data will not /// be checked against the given hash. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello world").await?; /// cacache::copy_hash_unchecked("./my-cache", &sri, "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn copy_hash_unchecked(cache: P, sri: &Integrity, to: Q) -> Result where P: AsRef, Q: AsRef, { read::copy_unchecked_async(cache.as_ref(), sri, to.as_ref()).await } /// Creates a reflink/clonefile from a cache entry to a destination path. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::reflink("./my-cache", "my-key", "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn reflink(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { async fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find_async(cache, key).await? { reflink_hash(cache, &entry.integrity, to).await } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()).await } /// Reflinks/clonefiles cache data to a specified location. Cache data will /// not be checked during linking. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::reflink_unchecked("./my-cache", "my-key", "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn reflink_unchecked(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { async fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find_async(cache, key).await? { reflink_hash_unchecked_sync(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()).await } /// Reflinks/clonefiles cache data by hash to a specified location. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello world").await?; /// cacache::reflink_hash("./my-cache", &sri, "./data.txt").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn reflink_hash(cache: P, sri: &Integrity, to: Q) -> Result<()> where P: AsRef, Q: AsRef, { read::reflink_async(cache.as_ref(), sri, to.as_ref()).await } /// Hard links a cache entry by key to a specified location. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn hard_link(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { async fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find(cache, key)? { read::hard_link_async(cache, &entry.integrity, to).await } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()).await } /// Gets the metadata entry for a certain key. /// /// Note that the existence of a metadata entry is not a guarantee that the /// underlying data exists, since they are stored and managed independently. /// To verify that the underlying associated data exists, use `exists()`. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn metadata(cache: P, key: K) -> Result> where P: AsRef, K: AsRef, { index::find_async(cache.as_ref(), key.as_ref()).await } /// Returns true if the given hash exists in the cache. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn exists>(cache: P, sri: &Integrity) -> bool { read::has_content_async(cache.as_ref(), sri).await.is_some() } // --------------- // Synchronous API // --------------- /// File handle for reading data synchronously. /// /// Make sure to call `get.check()` when done reading /// to verify that the extracted data passes integrity /// verification. pub struct SyncReader { reader: read::Reader, } impl std::io::Read for SyncReader { fn read(&mut self, buf: &mut [u8]) -> std::io::Result { self.reader.read(buf) } } impl SyncReader { /// Checks that data read from disk passes integrity checks. Returns the /// algorithm that was used verified the data. Should be called only after /// all data has been read from disk. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let mut fd = cacache::SyncReader::open("./my-cache", "my-key")?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).expect("Failed to read to string"); /// // Remember to check that the data you got was correct! /// fd.check()?; /// Ok(()) /// } /// ``` pub fn check(self) -> Result { self.reader.check() } /// Opens a new synchronous file handle into the cache, looking it up in the /// index using `key`. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let mut fd = cacache::SyncReader::open("./my-cache", "my-key")?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).expect("Failed to parse string"); /// // Remember to check that the data you got was correct! /// fd.check()?; /// Ok(()) /// } /// ``` pub fn open(cache: P, key: K) -> Result where P: AsRef, K: AsRef, { fn inner(cache: &Path, key: &str) -> Result { if let Some(entry) = index::find(cache, key)? { SyncReader::open_hash(cache, entry.integrity) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref()) } /// Opens a new synchronous file handle into the cache, based on its integrity address. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "key", b"hello world")?; /// let mut fd = cacache::SyncReader::open_hash("./my-cache", sri)?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).expect("Failed to read to string"); /// // Remember to check that the data you got was correct! /// fd.check()?; /// Ok(()) /// } /// ``` pub fn open_hash

(cache: P, sri: Integrity) -> Result where P: AsRef, { Ok(SyncReader { reader: read::open(cache.as_ref(), sri)?, }) } } /// Reads the entire contents of a cache file synchronously into a bytes /// vector, looking the data up by key. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let data = cacache::read_sync("./my-cache", "my-key")?; /// Ok(()) /// } /// ``` pub fn read_sync(cache: P, key: K) -> Result> where P: AsRef, K: AsRef, { fn inner(cache: &Path, key: &str) -> Result> { if let Some(entry) = index::find(cache, key)? { read_hash_sync(cache, &entry.integrity) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref()) } /// Reads the entire contents of a cache file synchronously into a bytes /// vector, looking the data up by its content address. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// let data = cacache::read_hash_sync("./my-cache", &sri)?; /// Ok(()) /// } /// ``` pub fn read_hash_sync

(cache: P, sri: &Integrity) -> Result> where P: AsRef, { read::read(cache.as_ref(), sri) } /// Copies a cache entry by key to a specified location. Returns the number of /// bytes copied. /// /// On platforms that support it, this will create a copy-on-write "reflink" /// with a full-copy fallback. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// cacache::copy_sync("./my-cache", "my-key", "./my-hello.txt")?; /// Ok(()) /// } /// ``` pub fn copy_sync(cache: P, key: K, to: Q) -> Result where P: AsRef, K: AsRef, Q: AsRef, { fn inner(cache: &Path, key: &str, to: &Path) -> Result { if let Some(entry) = index::find(cache, key)? { copy_hash_sync(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()) } /// Copies a cache entry by key to a specified location. Does not verify cache /// contents while copying. /// /// On platforms that support it, this will create a copy-on-write "reflink" /// with a full-copy fallback. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// cacache::copy_unchecked_sync("./my-cache", "my-key", "./my-hello.txt")?; /// Ok(()) /// } /// ``` pub fn copy_unchecked_sync(cache: P, key: K, to: Q) -> Result where P: AsRef, K: AsRef, Q: AsRef, { fn inner(cache: &Path, key: &str, to: &Path) -> Result { if let Some(entry) = index::find(cache, key)? { copy_hash_unchecked_sync(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()) } /// Copies a cache entry by integrity address to a specified location. Returns /// the number of bytes copied. /// /// On platforms that support it, this will create a copy-on-write "reflink" /// with a full-copy fallback. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// cacache::copy_hash_sync("./my-cache", &sri, "./my-hello.txt")?; /// Ok(()) /// } /// ``` pub fn copy_hash_sync(cache: P, sri: &Integrity, to: Q) -> Result where P: AsRef, Q: AsRef, { read::copy(cache.as_ref(), sri, to.as_ref()) } /// Copies a cache entry by integrity address to a specified location. Does /// not verify cache contents while copying. /// /// On platforms that support it, this will create a copy-on-write "reflink" /// with a full-copy fallback. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// cacache::copy_hash_unchecked_sync("./my-cache", &sri, "./my-hello.txt")?; /// Ok(()) /// } /// ``` pub fn copy_hash_unchecked_sync(cache: P, sri: &Integrity, to: Q) -> Result where P: AsRef, Q: AsRef, { read::copy_unchecked(cache.as_ref(), sri, to.as_ref()) } /// Creates a reflink/clonefile from a cache entry to a destination path. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::reflink_sync("./my-cache", "my-key", "./data.txt")?; /// Ok(()) /// } /// ``` pub fn reflink_sync(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find(cache, key)? { reflink_hash_sync(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()) } /// Reflinks/clonefiles cache data by hash to a specified location. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello world")?; /// cacache::reflink_hash_sync("./my-cache", &sri, "./data.txt")?; /// Ok(()) /// } /// ``` pub fn reflink_hash_sync(cache: P, sri: &Integrity, to: Q) -> Result<()> where P: AsRef, Q: AsRef, { read::reflink(cache.as_ref(), sri, to.as_ref()) } /// Reflinks/clonefiles cache data by hash to a specified location. Cache data /// will not be checked during linking. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello world")?; /// cacache::reflink_hash_unchecked_sync("./my-cache", &sri, "./data.txt")?; /// Ok(()) /// } /// ``` pub fn reflink_hash_unchecked_sync(cache: P, sri: &Integrity, to: Q) -> Result<()> where P: AsRef, Q: AsRef, { read::reflink_unchecked(cache.as_ref(), sri, to.as_ref()) } /// Reflinks/clonefiles cache data to a specified location. Cache data will /// not be checked during linking. /// /// Fails if the destination is on a different filesystem or if the filesystem /// does not support reflinks. /// /// Currently, reflinks are known to work on APFS (macOS), XFS, btrfs, and /// ReFS (Windows DevDrive) /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::reflink_unchecked_sync("./my-cache", "my-key", "./data.txt")?; /// Ok(()) /// } /// ``` pub fn reflink_unchecked_sync(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find(cache, key)? { reflink_hash_unchecked_sync(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()) } /// Hard links a cache entry by key to a specified location. The cache entry /// contents will not be checked, and all the usual caveats of hard links /// apply: The potentially-shared cache might be corrupted if the hard link is /// modified. pub fn hard_link_unchecked_sync(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find(cache, key)? { hard_link_hash_unchecked_sync(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()) } /// Hard links a cache entry by key to a specified location. pub fn hard_link_sync(cache: P, key: K, to: Q) -> Result<()> where P: AsRef, K: AsRef, Q: AsRef, { fn inner(cache: &Path, key: &str, to: &Path) -> Result<()> { if let Some(entry) = index::find(cache, key)? { read::hard_link(cache, &entry.integrity, to) } else { Err(Error::EntryNotFound(cache.to_path_buf(), key.into())) } } inner(cache.as_ref(), key.as_ref(), to.as_ref()) } /// Hard links a cache entry by integrity address to a specified location, /// verifying contents as hard links are created. pub fn hard_link_hash_sync(cache: P, sri: &Integrity, to: Q) -> Result<()> where P: AsRef, Q: AsRef, { read::hard_link(cache.as_ref(), sri, to.as_ref()) } /// Hard links a cache entry by integrity address to a specified location. The /// cache entry contents will not be checked, and all the usual caveats of /// hard links apply: The potentially-shared cache might be corrupted if the /// hard link is modified. pub fn hard_link_hash_unchecked_sync(cache: P, sri: &Integrity, to: Q) -> Result<()> where P: AsRef, Q: AsRef, { read::hard_link_unchecked(cache.as_ref(), sri, to.as_ref()) } /// Gets metadata for a certain key. /// /// Note that the existence of a metadata entry is not a guarantee that the /// underlying data exists, since they are stored and managed independently. /// To verify that the underlying associated data exists, use `exists_sync()`. pub fn metadata_sync(cache: P, key: K) -> Result> where P: AsRef, K: AsRef, { index::find(cache.as_ref(), key.as_ref()) } /// Returns true if the given hash exists in the cache. pub fn exists_sync>(cache: P, sri: &Integrity) -> bool { read::has_content(cache.as_ref(), sri).is_some() } #[cfg(test)] mod tests { #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncReadExt; use std::fs; #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_open() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::write(&dir, "my-key", b"hello world").await.unwrap(); let mut handle = crate::Reader::open(&dir, "my-key").await.unwrap(); let mut str = String::new(); handle.read_to_string(&mut str).await.unwrap(); handle.check().unwrap(); assert_eq!(str, String::from("hello world")); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_open_hash() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write(&dir, "my-key", b"hello world").await.unwrap(); let mut handle = crate::Reader::open_hash(&dir, sri).await.unwrap(); let mut str = String::new(); handle.read_to_string(&mut str).await.unwrap(); handle.check().unwrap(); assert_eq!(str, String::from("hello world")); } #[test] fn test_open_sync() { use std::io::prelude::*; let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::write_sync(&dir, "my-key", b"hello world").unwrap(); let mut handle = crate::SyncReader::open(&dir, "my-key").unwrap(); let mut str = String::new(); handle.read_to_string(&mut str).unwrap(); handle.check().unwrap(); assert_eq!(str, String::from("hello world")); } #[test] fn test_open_hash_sync() { use std::io::prelude::*; let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write_sync(&dir, "my-key", b"hello world").unwrap(); let mut handle = crate::SyncReader::open_hash(&dir, sri).unwrap(); let mut str = String::new(); handle.read_to_string(&mut str).unwrap(); handle.check().unwrap(); assert_eq!(str, String::from("hello world")); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_read() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::write(&dir, "my-key", b"hello world").await.unwrap(); let data = crate::read(&dir, "my-key").await.unwrap(); assert_eq!(data, b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_read_hash() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write(&dir, "my-key", b"hello world").await.unwrap(); let data = crate::read_hash(&dir, &sri).await.unwrap(); assert_eq!(data, b"hello world"); } #[test] fn test_read_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::write_sync(&dir, "my-key", b"hello world").unwrap(); let data = crate::read_sync(&dir, "my-key").unwrap(); assert_eq!(data, b"hello world"); } #[test] fn test_read_hash_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write_sync(&dir, "my-key", b"hello world").unwrap(); let data = crate::read_hash_sync(&dir, &sri).unwrap(); assert_eq!(data, b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_copy() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path(); let dest = dir.join("data"); crate::write(&dir, "my-key", b"hello world").await.unwrap(); crate::copy(&dir, "my-key", &dest).await.unwrap(); let data = crate::async_lib::read(&dest).await.unwrap(); assert_eq!(data, b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_copy_hash() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path(); let dest = dir.join("data"); let sri = crate::write(&dir, "my-key", b"hello world").await.unwrap(); crate::copy_hash(&dir, &sri, &dest).await.unwrap(); let data = crate::async_lib::read(&dest).await.unwrap(); assert_eq!(data, b"hello world"); } #[test] fn test_copy_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path(); let dest = dir.join("data"); crate::write_sync(dir, "my-key", b"hello world").unwrap(); crate::copy_sync(dir, "my-key", &dest).unwrap(); let data = fs::read(&dest).unwrap(); assert_eq!(data, b"hello world"); } #[test] fn test_copy_hash_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path(); let dest = dir.join("data"); let sri = crate::write_sync(dir, "my-key", b"hello world").unwrap(); crate::copy_hash_sync(dir, &sri, &dest).unwrap(); let data = fs::read(&dest).unwrap(); assert_eq!(data, b"hello world"); } } cacache-13.0.0/src/index.rs000064400000000000000000000550501046102023000135250ustar 00000000000000//! Raw access to the cache index. Use with caution! use std::collections::HashSet; use std::fs::{self, OpenOptions}; use std::hash::{Hash, Hasher}; use std::io::{ErrorKind, Write}; use std::path::{Path, PathBuf}; use std::time::{SystemTime, UNIX_EPOCH}; use digest::Digest; use either::{Left, Right}; #[cfg(any(feature = "async-std", feature = "tokio"))] use futures::stream::StreamExt; use serde_derive::{Deserialize, Serialize}; use serde_json::Value; use sha1::Sha1; use sha2::Sha256; use ssri::Integrity; use walkdir::WalkDir; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::{AsyncBufReadExt, AsyncWriteExt}; use crate::content::path::content_path; use crate::errors::{IoErrorExt, Result}; use crate::put::WriteOpts; const INDEX_VERSION: &str = "5"; /// Represents a cache index entry, which points to content. #[derive(PartialEq, Debug)] pub struct Metadata { /// Key this entry is stored under. pub key: String, /// Integrity hash for the stored data. Acts as a key into {cache}/content. pub integrity: Integrity, /// Timestamp in unix milliseconds when this entry was written. pub time: u128, /// Size of data associated with this entry. pub size: usize, /// Arbitrary JSON associated with this entry. pub metadata: Value, /// Raw metadata in binary form. Can be different from JSON metadata. pub raw_metadata: Option>, } #[derive(Deserialize, Serialize, Debug)] struct SerializableMetadata { key: String, integrity: Option, time: u128, size: usize, metadata: Value, raw_metadata: Option>, } impl PartialEq for SerializableMetadata { fn eq(&self, other: &Self) -> bool { self.key == other.key } } impl Eq for SerializableMetadata {} impl Hash for SerializableMetadata { fn hash(&self, state: &mut H) { self.key.hash(state); } } /// Raw insertion into the cache index. pub fn insert(cache: &Path, key: &str, opts: WriteOpts) -> Result { let bucket = bucket_path(cache, key); fs::create_dir_all(bucket.parent().unwrap()).with_context(|| { format!( "Failed to create index bucket directory: {:?}", bucket.parent().unwrap() ) })?; let stringified = serde_json::to_string(&SerializableMetadata { key: key.to_owned(), integrity: opts.sri.clone().map(|x| x.to_string()), time: opts.time.unwrap_or_else(now), size: opts.size.unwrap_or(0), metadata: opts.metadata.unwrap_or(serde_json::Value::Null), raw_metadata: opts.raw_metadata, }) .with_context(|| format!("Failed to serialize entry with key `{key}`"))?; let mut buck = OpenOptions::new() .create(true) .append(true) .open(&bucket) .with_context(|| format!("Failed to create or open index bucket at {bucket:?}"))?; let out = format!("\n{}\t{}", hash_entry(&stringified), stringified); buck.write_all(out.as_bytes()) .with_context(|| format!("Failed to write to index bucket at {bucket:?}"))?; buck.flush() .with_context(|| format!("Failed to flush bucket at {bucket:?}"))?; Ok(opts .sri .or_else(|| "sha1-deadbeef".parse::().ok()) .unwrap()) } #[cfg(any(feature = "async-std", feature = "tokio"))] /// Asynchronous raw insertion into the cache index. pub async fn insert_async<'a>(cache: &'a Path, key: &'a str, opts: WriteOpts) -> Result { let bucket = bucket_path(cache, key); crate::async_lib::create_dir_all(bucket.parent().unwrap()) .await .with_context(|| { format!( "Failed to create index bucket directory: {:?}", bucket.parent().unwrap() ) })?; let stringified = serde_json::to_string(&SerializableMetadata { key: key.to_owned(), integrity: opts.sri.clone().map(|x| x.to_string()), time: opts.time.unwrap_or_else(now), size: opts.size.unwrap_or(0), metadata: opts.metadata.unwrap_or(serde_json::Value::Null), raw_metadata: opts.raw_metadata, }) .with_context(|| format!("Failed to serialize entry with key `{key}`"))?; let mut buck = crate::async_lib::OpenOptions::new() .create(true) .append(true) .open(&bucket) .await .with_context(|| format!("Failed to create or open index bucket at {bucket:?}"))?; let out = format!("\n{}\t{}", hash_entry(&stringified), stringified); buck.write_all(out.as_bytes()) .await .with_context(|| format!("Failed to write to index bucket at {bucket:?}"))?; buck.flush() .await .with_context(|| format!("Failed to flush bucket at {bucket:?}"))?; Ok(opts .sri .or_else(|| "sha1-deadbeef".parse::().ok()) .unwrap()) } /// Raw index Metadata access. pub fn find(cache: &Path, key: &str) -> Result> { let bucket = bucket_path(cache, key); Ok(bucket_entries(&bucket) .with_context(|| format!("Failed to read index bucket entries from {bucket:?}"))? .into_iter() .fold(None, |acc, entry| { if entry.key == key { if let Some(integrity) = entry.integrity { let integrity: Integrity = match integrity.parse() { Ok(sri) => sri, _ => return acc, }; Some(Metadata { key: entry.key, integrity, size: entry.size, time: entry.time, metadata: entry.metadata, raw_metadata: entry.raw_metadata, }) } else { None } } else { acc } })) } #[cfg(any(feature = "async-std", feature = "tokio"))] /// Asynchronous raw index Metadata access. pub async fn find_async(cache: &Path, key: &str) -> Result> { let bucket = bucket_path(cache, key); Ok(bucket_entries_async(&bucket) .await .with_context(|| format!("Failed to read index bucket entries from {bucket:?}"))? .into_iter() .fold(None, |acc, entry| { if entry.key == key { if let Some(integrity) = entry.integrity { let integrity: Integrity = match integrity.parse() { Ok(sri) => sri, _ => return acc, }; Some(Metadata { key: entry.key, integrity, size: entry.size, time: entry.time, metadata: entry.metadata, raw_metadata: entry.raw_metadata, }) } else { None } } else { acc } })) } /// Deletes an index entry, without deleting the actual cache data entry. pub fn delete(cache: &Path, key: &str) -> Result<()> { insert( cache, key, WriteOpts { algorithm: None, size: None, sri: None, time: None, metadata: None, raw_metadata: None, }, ) .map(|_| ()) } #[cfg(any(feature = "async-std", feature = "tokio"))] /// Asynchronously deletes an index entry, without deleting the actual cache /// data entry. pub async fn delete_async(cache: &Path, key: &str) -> Result<()> { insert( cache, key, WriteOpts { algorithm: None, size: None, sri: None, time: None, metadata: None, raw_metadata: None, }, ) .map(|_| ()) } /// Lists raw index Metadata entries. pub fn ls(cache: &Path) -> impl Iterator> { let cache_path = cache.join(format!("index-v{INDEX_VERSION}")); let cloned = cache_path.clone(); WalkDir::new(&cache_path) .into_iter() .map(move |bucket| { let bucket = bucket .map_err(|e| match e.io_error() { Some(io_err) => std::io::Error::new(io_err.kind(), io_err.kind().to_string()), None => crate::errors::io_error("Unexpected error"), }) .with_context(|| { format!( "Error while walking cache index directory at {}", cloned.display() ) })?; if bucket.file_type().is_dir() { return Ok(Vec::new()); } let owned_path = bucket.path().to_owned(); Ok(bucket_entries(bucket.path()) .with_context(|| { format!("Error getting bucket entries from {}", owned_path.display()) })? .into_iter() .rev() .collect::>() .into_iter() .filter_map(|se| { if let Some(i) = se.integrity { Some(Metadata { key: se.key, integrity: i.parse().unwrap(), time: se.time, size: se.size, metadata: se.metadata, raw_metadata: se.raw_metadata, }) } else { None } }) .collect()) }) .flat_map(|res| match res { Ok(it) => Left(it.into_iter().map(Ok)), Err(err) => Right(std::iter::once(Err(err))), }) } fn bucket_path(cache: &Path, key: &str) -> PathBuf { let hashed = hash_key(key); cache .join(format!("index-v{INDEX_VERSION}")) .join(&hashed[0..2]) .join(&hashed[2..4]) .join(&hashed[4..]) } fn hash_key(key: &str) -> String { let mut hasher = Sha1::new(); hasher.update(key); hex::encode(hasher.finalize()) } fn hash_entry(key: &str) -> String { let mut hasher = Sha256::new(); hasher.update(key); hex::encode(hasher.finalize()) } fn now() -> u128 { SystemTime::now() .duration_since(UNIX_EPOCH) .unwrap() .as_millis() } fn bucket_entries(bucket: &Path) -> std::io::Result> { use std::io::{BufRead, BufReader}; fs::File::open(bucket) .map(|file| { BufReader::new(file) .lines() .map_while(std::result::Result::ok) .filter_map(|entry| { let entry_str = match entry.split('\t').collect::>()[..] { [hash, entry_str] if hash_entry(entry_str) == hash => entry_str, // Something's wrong with the entry. Abort. _ => return None, }; serde_json::from_str::(entry_str).ok() }) .collect() }) .or_else(|err| { if err.kind() == ErrorKind::NotFound { Ok(Vec::new()) } else { Err(err)? } }) } #[cfg(any(feature = "async-std", feature = "tokio"))] async fn bucket_entries_async(bucket: &Path) -> std::io::Result> { let file_result = crate::async_lib::File::open(bucket).await; let file = if let Err(err) = file_result { if err.kind() == ErrorKind::NotFound { return Ok(Vec::new()); } return Err(err)?; } else { file_result.unwrap() }; let mut vec = Vec::new(); let mut lines = crate::async_lib::lines_to_stream(crate::async_lib::BufReader::new(file).lines()); while let Some(line) = lines.next().await { if let Ok(entry) = line { let entry_str = match entry.split('\t').collect::>()[..] { [hash, entry_str] if hash_entry(entry_str) == hash => entry_str, // Something's wrong with the entry. Abort. _ => continue, }; if let Ok(serialized) = serde_json::from_str::(entry_str) { vec.push(serialized); } } } Ok(vec) } /// Builder for options and flags for remove cache entry. #[derive(Clone, Default)] pub struct RemoveOpts { pub(crate) remove_fully: bool, } impl RemoveOpts { /// Creates cache remove options. pub fn new() -> Self { Default::default() } /// Set the remove fully option /// If remove_fully is set to true then the index file itself will be physically deleted rather than appending a null. pub fn remove_fully(mut self, remove_fully: bool) -> Self { self.remove_fully = remove_fully; self } /// Removes an individual index metadata entry. The associated content will be left in the cache. pub fn remove_sync(self, cache: P, key: K) -> Result<()> where P: AsRef, K: AsRef, { if !self.remove_fully { delete(cache.as_ref(), key.as_ref()) } else { if let Some(meta) = crate::metadata_sync(cache.as_ref(), key.as_ref())? { let content = content_path(cache.as_ref(), &meta.integrity); fs::remove_file(&content) .with_context(|| format!("Failed to remove content at {content:?}"))?; } let bucket = bucket_path(cache.as_ref(), key.as_ref()); fs::remove_file(&bucket) .with_context(|| format!("Failed to remove bucket at {bucket:?}")) } } /// Removes an individual index metadata entry. The associated content will be left in the cache. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn remove(self, cache: P, key: K) -> Result<()> where P: AsRef, K: AsRef, { if !self.remove_fully { delete_async(cache.as_ref(), key.as_ref()).await } else { if let Some(meta) = crate::metadata(cache.as_ref(), key.as_ref()).await? { let content = content_path(cache.as_ref(), &meta.integrity); crate::async_lib::remove_file(&content) .await .with_context(|| format!("Failed to remove content at {content:?}"))?; } let bucket = bucket_path(cache.as_ref(), key.as_ref()); crate::async_lib::remove_file(&bucket) .await .with_context(|| format!("Failed to remove bucket at {bucket:?}")) } } } #[cfg(test)] mod tests { use super::*; use serde_json::json; #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; const MOCK_ENTRY: &str = "\n9cbbfe2553e7c7e1773f53f0f643fdd72008faa38da53ebcb055e5e20321ae47\t{\"key\":\"hello\",\"integrity\":\"sha1-deadbeef\",\"time\":1234567,\"size\":0,\"metadata\":null,\"raw_metadata\":null}"; fn ls_entries(dir: &Path) -> Vec { let mut entries = ls(dir) .map(|x| Ok(x?.key)) .collect::>>() .unwrap(); entries.sort(); entries } #[test] fn insert_basic() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri).time(time); insert(&dir, "hello", opts).unwrap(); let entry = std::fs::read_to_string(bucket_path(&dir, "hello")).unwrap(); assert_eq!(entry, MOCK_ENTRY); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn insert_async_basic() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri).time(time); futures::executor::block_on(async { insert_async(&dir, "hello", opts).await.unwrap(); }); let entry = std::fs::read_to_string(bucket_path(&dir, "hello")).unwrap(); assert_eq!(entry, MOCK_ENTRY); } #[test] fn find_basic() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let bucket = bucket_path(&dir, "hello"); fs::create_dir_all(bucket.parent().unwrap()).unwrap(); fs::write(bucket, MOCK_ENTRY).unwrap(); let entry = find(&dir, "hello").unwrap().unwrap(); assert_eq!( entry, Metadata { key: String::from("hello"), integrity: sri, time, size: 0, metadata: json!(null), raw_metadata: None, } ); } #[test] fn find_none() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); assert_eq!(find(&dir, "hello").unwrap(), None); } #[test] fn delete_basic() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri).time(time); insert(&dir, "hello", opts).unwrap(); delete(&dir, "hello").unwrap(); assert_eq!(find(&dir, "hello").unwrap(), None); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn delete_async_basic() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri).time(time); insert(&dir, "hello", opts).unwrap(); futures::executor::block_on(async { delete_async(&dir, "hello").await.unwrap(); }); assert_eq!(find(&dir, "hello").unwrap(), None); } #[test] fn delete_fully() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let content = content_path(&dir, &"sha1-deadbeef".parse().unwrap()); fs::create_dir_all(content.parent().unwrap()).unwrap(); fs::write(content.as_path(), "hello").unwrap(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; insert(&dir, "hello", WriteOpts::new().integrity(sri).time(time)).unwrap(); RemoveOpts::new() .remove_fully(true) .remove_sync(&dir, "hello") .unwrap(); assert_eq!(find(&dir, "hello").unwrap(), None); assert!(!content.exists()); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn delete_fully_async() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let content = content_path(&dir, &"sha1-deadbeef".parse().unwrap()); fs::create_dir_all(content.parent().unwrap()).unwrap(); fs::write(content.as_path(), "hello").unwrap(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; insert(&dir, "hello", WriteOpts::new().integrity(sri).time(time)).unwrap(); RemoveOpts::new() .remove_fully(true) .remove(&dir, "hello") .await .unwrap(); assert_eq!(find(&dir, "hello").unwrap(), None); assert!(!content.exists()); } #[test] fn round_trip() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri.clone()).time(time); insert(&dir, "hello", opts).unwrap(); let entry = find(&dir, "hello").unwrap().unwrap(); assert_eq!( entry, Metadata { key: String::from("hello"), integrity: sri, time, size: 0, metadata: json!(null), raw_metadata: None, } ); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn round_trip_async() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri.clone()).time(time); futures::executor::block_on(async { insert_async(&dir, "hello", opts).await.unwrap(); }); let entry = futures::executor::block_on(async { find_async(&dir, "hello").await.unwrap().unwrap() }); assert_eq!( entry, Metadata { key: String::from("hello"), integrity: sri, time, size: 0, metadata: json!(null), raw_metadata: None, } ); } #[test] fn ls_basic() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri.clone()).time(time); insert(&dir, "hello", opts).unwrap(); let opts = WriteOpts::new().integrity(sri).time(time); insert(&dir, "world", opts).unwrap(); let entries = ls_entries(&dir); assert_eq!(entries, vec![String::from("hello"), String::from("world")]) } #[test] fn ls_basic_with_delete() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri: Integrity = "sha1-deadbeef".parse().unwrap(); let time = 1_234_567; let opts = WriteOpts::new().integrity(sri.clone()).time(time); insert(&dir, "hello", opts).unwrap(); let opts = WriteOpts::new().integrity(sri).time(time); insert(&dir, "world", opts).unwrap(); let entries = ls_entries(&dir); assert_eq!(entries, vec![String::from("hello"), String::from("world")]); delete(&dir, "hello").unwrap(); let entries = ls_entries(&dir); assert_eq!(entries, vec![String::from("world")]) } } cacache-13.0.0/src/lib.rs000064400000000000000000000140241046102023000131600ustar 00000000000000//! cacache is a Rust library for managing local key and content address //! caches. It's really fast, really good at concurrency, and it will never //! give you corrupted data, even if cache files get corrupted or manipulated. //! //! ## API Layout //! //! The cacache API is organized roughly similar to `std::fs`; most of the //! toplevel functionality is available as free functions directly in the //! `cacache` module, with some additional functionality available through //! returned objects, as well as `WriteOpts`, which is analogous to //! `OpenOpts`, but is only able to write. //! //! One major difference is that the default APIs are all async functions, as //! opposed to `std::fs`, where they're all synchronous. Synchronous APIs in //! cacache are accessible through the `_sync` suffix. //! //! ### Suffixes //! //! You may notice various suffixes associated with otherwise familiar //! functions: //! //! * `_sync` - Most cacache APIs are asynchronous by default. Anything using //! the `_sync` suffix behaves just like its unprefixed counterpart, except //! the operation is synchronous. //! * `_hash` - Since cacache is a content-addressable cache, the `_hash` //! suffix means you're interacting directly with content data, skipping the //! index and its metadata. These functions use an `Integrity` to look up //! data, instead of a string key. //! //! ## Examples //! //! Un-suffixed APIs are all async, using //! [`async-std`](https://crates.io/crates/async-std). They let you put data //! in and get it back out -- asynchronously! //! //! ```no_run //! use async_attributes; //! //! #[async_attributes::main] //! async fn main() -> cacache::Result<()> { //! // Data goes in... //! cacache::write("./my-cache", "key", b"hello").await?; //! //! // ...data comes out! //! let data = cacache::read("./my-cache", "key").await?; //! assert_eq!(data, b"hello"); //! //! Ok(()) //! } //! ``` //! //! ### Lookup by hash //! //! What makes `cacache` content addressable, though, is its ability to fetch //! data by its "content address", which in our case is a ["subresource //! integrity" hash](https://crates.io/crates/ssri), which `cacache::put` //! conveniently returns for us. Fetching data by hash is significantly faster //! than doing key lookups: //! //! ```no_run //! use async_attributes; //! //! #[async_attributes::main] //! async fn main() -> cacache::Result<()> { //! // Data goes in... //! let sri = cacache::write("./my-cache", "key", b"hello").await?; //! //! // ...data gets looked up by `sri` ("Subresource Integrity"). //! let data = cacache::read_hash("./my-cache", &sri).await?; //! assert_eq!(data, b"hello"); //! //! Ok(()) //! } //! ``` //! //! ### Large file support //! //! `cacache` supports large file reads, in both async and sync mode, through //! an API reminiscent of `std::fs::OpenOptions`: //! //! ```no_run //! use async_attributes; //! use async_std::prelude::*; //! //! #[async_attributes::main] //! async fn main() -> cacache::Result<()> { //! let mut fd = cacache::Writer::create("./my-cache", "key").await?; //! for _ in 0..10 { //! fd.write_all(b"very large data").await.expect("Failed to write to cache"); //! } //! // Data is only committed to the cache after you do `fd.commit()`! //! let sri = fd.commit().await?; //! println!("integrity: {}", &sri); //! //! let mut fd = cacache::Reader::open("./my-cache", "key").await?; //! let mut buf = String::new(); //! fd.read_to_string(&mut buf).await.expect("Failed to read to string"); //! //! // Make sure to call `.check()` when you're done! It makes sure that what //! // you just read is actually valid. `cacache` always verifies the data //! // you get out is what it's supposed to be. The check is very cheap! //! fd.check()?; //! //! Ok(()) //! } //! ``` //! //! ### Sync API //! //! There are also sync APIs available if you don't want to use async/await. //! The synchronous APIs are generally faster for linear operations -- that //! is, doing one thing after another, as opposed to doing many things at //! once. If you're only reading and writing one thing at a time across your //! application, you probably want to use these instead. //! //! If you wish to _only_ use sync APIs and not pull in an async runtime, you //! can disable default features: //! //! ```toml //! # Cargo.toml //! [dependencies] //! cacache = { version = "X.Y.Z", default-features = false, features = ["mmap"] } //! ``` //! //! ```no_run //! fn main() -> cacache::Result<()> { //! cacache::write_sync("./my-cache", "key", b"my-data").unwrap(); //! let data = cacache::read_sync("./my-cache", "key").unwrap(); //! assert_eq!(data, b"my-data"); //! Ok(()) //! } //! ``` //! //! ### Linking to existing files //! //! The `link_to` feature enables an additional set of APIs for adding //! existing files into the cache via symlinks, without having to duplicate //! their data. Once the cache links to them, these files can be accessed by //! key just like other cached data, with the same integrity checking. //! //! The `link_to` methods are available in both async and sync variants, using //! the same suffixes as the other APIs. //! //! ```no_run //! #[async_attributes::main] //! async fn main() -> cacache::Result<()> { //! #[cfg(feature = "link_to")] //! cacache::link_to("./my-cache", "key", "/path/to/my-other-file.txt").await?; //! let data = cacache::read("./my-cache", "key").await?; //! assert_eq!(data, b"my-data"); //! Ok(()) //! } //! ``` #![warn(missing_docs)] #[cfg(all(feature = "async-std", feature = "tokio-runtime"))] compile_error!("Only either feature \"async-std\" or \"tokio-runtime\" must be enabled for this crate, not both."); pub use serde_json::Value; pub use ssri::{Algorithm, Integrity}; #[cfg(any(feature = "async-std", feature = "tokio"))] mod async_lib; mod content; mod errors; pub mod index; mod get; #[cfg(feature = "link_to")] mod linkto; mod ls; mod put; mod rm; pub use errors::{Error, Result}; pub use index::{Metadata, RemoveOpts}; pub use get::*; #[cfg(feature = "link_to")] pub use linkto::*; pub use ls::*; pub use put::*; pub use rm::*; cacache-13.0.0/src/linkto.rs000064400000000000000000000505401046102023000137150ustar 00000000000000#[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncRead; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::AsyncReadExt; use crate::content::linkto; use crate::errors::{Error, IoErrorExt, Result}; use crate::{index, WriteOpts}; use ssri::{Algorithm, Integrity}; use std::io::Read; use std::path::{Path, PathBuf}; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::pin::Pin; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::task::{Context as TaskContext, Poll}; const BUF_SIZE: usize = 16 * 1024; const PROBE_SIZE: usize = 8; /// Asynchronously adds `target` to the `cache` with a symlink, indexing it /// under `key`. /// /// ## Example /// ```no_run /// use async_attributes; /// use std::path::Path; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::link_to("./my-cache", "my-key", "../my-other-files/my-file.tgz").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn link_to(cache: P, key: K, target: T) -> Result where P: AsRef, K: AsRef, T: AsRef, { ToLinker::open(cache, key, target).await?.commit().await } /// Asynchrounously adds `target` to the `cache` with a symlink, skipping /// associating an index key with it. /// /// ## Example /// ```no_run /// use async_attributes; /// use std::path::Path; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::link_to_hash("./my-cache", "../my-other-files/my-file.tgz").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn link_to_hash(cache: P, target: T) -> Result where P: AsRef, T: AsRef, { ToLinker::open_hash(cache, target).await?.commit().await } /// Synchronously creates a symlink in the `cache` to the `target`, indexing it /// under `key`. /// /// ## Example /// ```no_run /// use std::io::Read; /// use std::path::Path; /// /// fn main() -> cacache::Result<()> { /// cacache::link_to_sync("./my-cache", "my-key", "../my-other-files/my-file.tgz")?; /// Ok(()) /// } /// ``` pub fn link_to_sync(cache: P, key: K, target: T) -> Result where P: AsRef, K: AsRef, T: AsRef, { SyncToLinker::open(cache, key, target)?.commit() } /// Synchronously creates a symlink in the `cache` to the `target`, skipping /// associating an index key with it. /// /// ## Example /// ```no_run /// use std::io::Read; /// use std::path::Path; /// /// fn main() -> cacache::Result<()> { /// cacache::link_to_hash_sync("./my-cache", "../foo/bar.tgz")?; /// Ok(()) /// } /// ``` pub fn link_to_hash_sync(cache: P, target: T) -> Result where P: AsRef, T: AsRef, { SyncToLinker::open_hash(cache, target)?.commit() } /// Extend the `WriteOpts` struct with factories for creating `ToLinker` and /// `SyncToLinker` instances. impl WriteOpts { /// Opens the target file handle for reading, returning a ToLinker instance. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn link_to(self, cache: P, key: K, target: T) -> Result where P: AsRef, K: AsRef, T: AsRef, { async fn inner( opts: WriteOpts, cache: &Path, key: &str, target: &Path, ) -> Result { Ok(ToLinker { cache: cache.to_path_buf(), key: Some(String::from(key)), read: 0, linker: linkto::AsyncToLinker::new( cache, opts.algorithm.unwrap_or(Algorithm::Sha256), target, ) .await?, opts, }) } inner(self, cache.as_ref(), key.as_ref(), target.as_ref()).await } /// Opens the target file handle for reading, without a key, returning a /// ToLinker instance. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn link_to_hash(self, cache: P, target: T) -> Result where P: AsRef, T: AsRef, { async fn inner(opts: WriteOpts, cache: &Path, target: &Path) -> Result { Ok(ToLinker { cache: cache.to_path_buf(), key: None, read: 0, linker: linkto::AsyncToLinker::new( cache, opts.algorithm.unwrap_or(Algorithm::Sha256), target, ) .await?, opts, }) } inner(self, cache.as_ref(), target.as_ref()).await } /// Opens the target file handle for reading synchronously, returning a /// SyncToLinker instance. pub fn link_to_sync(self, cache: P, key: K, target: T) -> Result where P: AsRef, K: AsRef, T: AsRef, { fn inner(opts: WriteOpts, cache: &Path, key: &str, target: &Path) -> Result { Ok(SyncToLinker { cache: cache.to_path_buf(), key: Some(String::from(key)), read: 0, linker: linkto::ToLinker::new( cache, opts.algorithm.unwrap_or(Algorithm::Sha256), target, )?, opts, }) } inner(self, cache.as_ref(), key.as_ref(), target.as_ref()) } /// Opens the target file handle for reading synchronously, without a key, /// returning a SyncToLinker instance. pub fn link_to_hash_sync(self, cache: P, target: T) -> Result where P: AsRef, T: AsRef, { fn inner(opts: WriteOpts, cache: &Path, target: &Path) -> Result { Ok(SyncToLinker { cache: cache.to_path_buf(), key: None, read: 0, linker: linkto::ToLinker::new( cache, opts.algorithm.unwrap_or(Algorithm::Sha256), target, )?, opts, }) } inner(self, cache.as_ref(), target.as_ref()) } } /// A file handle for asynchronously reading in data from a file to be added to /// the cache via a symlink to the target file. /// /// Make sure to call `.commit()` when done reading to actually add the file to /// the cache. #[cfg(any(feature = "async-std", feature = "tokio"))] pub struct ToLinker { cache: PathBuf, key: Option, read: usize, pub(crate) linker: linkto::AsyncToLinker, opts: WriteOpts, } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncRead for ToLinker { #[cfg(feature = "async-std")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>, buf: &mut [u8], ) -> Poll> { let amt = futures::ready!(Pin::new(&mut self.linker).poll_read(cx, buf))?; self.read += amt; Poll::Ready(Ok(amt)) } #[cfg(feature = "tokio")] fn poll_read( mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>, buf: &mut tokio::io::ReadBuf<'_>, ) -> Poll> { let pre_len = buf.filled().len(); futures::ready!(Pin::new(&mut self.linker).poll_read(cx, buf))?; self.read += buf.filled().len() - pre_len; Poll::Ready(Ok(())) } } fn filesize(target: &Path) -> Result { Ok(target .metadata() .with_context(|| format!("Failed to get metadata of {}", target.display()))? .len() as usize) } #[cfg(any(feature = "async-std", feature = "tokio"))] impl ToLinker { /// Creates a new asynchronous readable file handle into the cache. pub async fn open(cache: P, key: K, target: T) -> Result where P: AsRef, K: AsRef, T: AsRef, { async fn inner(cache: &Path, key: &str, target: &Path) -> Result { let size = filesize(target)?; WriteOpts::new() .algorithm(Algorithm::Sha256) .size(size) .link_to(cache, key, target) .await } inner(cache.as_ref(), key.as_ref(), target.as_ref()).await } /// Creates a new asynchronous readable file handle into the cache. pub async fn open_hash(cache: P, target: T) -> Result where P: AsRef, T: AsRef, { async fn inner(cache: &Path, target: &Path) -> Result { let size = filesize(target)?; WriteOpts::new() .algorithm(Algorithm::Sha256) .size(size) .link_to_hash(cache, target) .await } inner(cache.as_ref(), target.as_ref()).await } /// Consumes the rest of the file handle, creates an symlink into /// the cache, and creates index entries for the linked file. Also verifies /// data against `size` and `integrity` options, if provided. Must be called /// manually in order to complete the writing process, otherwise everything /// will be thrown out. pub async fn commit(mut self) -> Result { self.consume().await?; let linker_sri = self.linker.commit().await?; if let Some(sri) = &self.opts.sri { if sri.matches(&linker_sri).is_none() { return Err(ssri::Error::IntegrityCheckError(sri.clone(), linker_sri).into()); } } else { self.opts.sri = Some(linker_sri.clone()); } if let Some(size) = self.opts.size { if size != self.read { return Err(Error::SizeMismatch(size, self.read)); } } if let Some(key) = self.key { index::insert(&self.cache, &key, self.opts) } else { Ok(linker_sri) } } // "Consume" the remainder of the reader, so that the integrity is properly // calculated. async fn consume(&mut self) -> Result<()> { // Do a small 'test' read to avoid allocating a larger buffer if it // isn't necessary. let mut probe = [0; PROBE_SIZE]; if self.context_read(&mut probe).await? > 0 { // Make sure all the bytes are read so that the integrity is // properly calculated. let mut buf = [0; BUF_SIZE]; while self.context_read(&mut buf).await? > 0 {} } Ok(()) } async fn context_read(&mut self, buf: &mut [u8]) -> Result { AsyncReadExt::read(self, buf).await.with_context(|| { "Failed to read target file contents while calculating integrity".into() }) } } /// A file handle for synchronously reading data from a file to be added to the /// cache via a symlink. /// /// Make sure to call `.commit()` when done reading to actually add the file /// to the cache. pub struct SyncToLinker { cache: PathBuf, key: Option, read: usize, pub(crate) linker: linkto::ToLinker, opts: WriteOpts, } impl std::io::Read for SyncToLinker { fn read(&mut self, buf: &mut [u8]) -> std::io::Result { let amt = self.linker.read(buf)?; self.read += amt; Ok(amt) } } impl SyncToLinker { /// Creates a new readable file handle to a file the cache will link to, /// indexed at the provided key, on commit. /// /// It is not necessary to read any of the file before calling `.commit()`. /// /// ## Example /// ```no_run /// use std::io::prelude::*; /// /// fn main() -> cacache::Result<()> { /// let path = "../my-other-files/my-file.tgz"; /// let mut fd = cacache::SyncToLinker::open("./my-cache", "my-key", path)?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).expect("Failed to read to string"); /// // The file is not linked into the cache until you commit it. /// fd.commit()?; /// Ok(()) /// } /// ``` pub fn open(cache: P, key: K, target: T) -> Result where P: AsRef, K: AsRef, T: AsRef, { fn inner(cache: &Path, key: &str, target: &Path) -> Result { let size = filesize(target)?; WriteOpts::new() .algorithm(Algorithm::Sha256) .size(size) .link_to_sync(cache, key, target) } inner(cache.as_ref(), key.as_ref(), target.as_ref()) } /// Creates a new readable file handle to a file that the cache will link /// to, without an indexe key, on commit. /// /// It is not necessary to read any of the file before calling `.commit()`. /// /// ## Example /// ```no_run /// use std::io::prelude::*; /// /// fn main() -> cacache::Result<()> { /// let path = "../my-other-files/my-file.tgz"; /// let mut fd = cacache::SyncToLinker::open_hash("./my-cache", path)?; /// let mut str = String::new(); /// fd.read_to_string(&mut str).expect("Failed to read to string"); /// // The file is not linked into the cache until you commit it. /// fd.commit()?; /// Ok(()) /// } /// ``` pub fn open_hash(cache: P, target: T) -> Result where P: AsRef, T: AsRef, { fn inner(cache: &Path, target: &Path) -> Result { let size = filesize(target)?; WriteOpts::new() .algorithm(Algorithm::Sha256) .size(size) .link_to_hash_sync(cache, target) } inner(cache.as_ref(), target.as_ref()) } /// Consumes the rest of the file handle, creates a symlink to the file, and /// creates index entries for the linked file. Also verifies data against /// `size` and `integrity` options, if provided. Must be called manually in /// order to complete the writing process, otherwise everything will be /// thrown out. pub fn commit(mut self) -> Result { self.consume()?; let cache = self.cache; let linker_sri = self.linker.commit()?; if let Some(sri) = &self.opts.sri { if sri.matches(&linker_sri).is_none() { return Err(ssri::Error::IntegrityCheckError(sri.clone(), linker_sri).into()); } } else { self.opts.sri = Some(linker_sri.clone()); } if let Some(size) = self.opts.size { if size != self.read { return Err(Error::SizeMismatch(size, self.read)); } } if let Some(key) = self.key { index::insert(&cache, &key, self.opts) } else { Ok(linker_sri) } } fn consume(&mut self) -> Result<()> { // Do a small 'test' read to avoid allocating a larger buffer if it // isn't necessary. let mut probe = [0; PROBE_SIZE]; if self.context_read(&mut probe)? > 0 { // Make sure all the bytes are read so that the integrity is // properly calculated. let mut buf = [0; BUF_SIZE]; while self.context_read(&mut buf)? > 0 {} } Ok(()) } fn context_read(&mut self, buf: &mut [u8]) -> Result { self.read(buf).with_context(|| { "Failed to read target file contents while calculating integrity".into() }) } } #[cfg(test)] mod tests { use std::fs::File; use std::io::Write; use super::*; #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; fn create_tmpfile(tmp: &tempfile::TempDir, buf: &[u8]) -> PathBuf { let dir = tmp.path().to_owned(); let target = dir.join("target-file"); std::fs::create_dir_all(target.parent().unwrap()).unwrap(); let mut file = File::create(&target).unwrap(); file.write_all(buf).unwrap(); file.flush().unwrap(); target } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_link() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::link_to(&dir, "my-key", target).await.unwrap(); let buf = crate::read(&dir, "my-key").await.unwrap(); assert_eq!(buf, b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_link_to_hash() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::link_to_hash(&dir, target).await.unwrap(); let buf = crate::read_hash(&dir, &sri).await.unwrap(); assert_eq!(buf, b"hello world"); } #[test] fn test_link_to_sync() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::link_to_sync(&dir, "my-key", target).unwrap(); let buf = crate::read_sync(&dir, "my-key").unwrap(); assert_eq!(buf, b"hello world"); } #[test] fn test_link_to_hash_sync() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::link_to_hash_sync(&dir, target).unwrap(); let buf = crate::read_hash_sync(&dir, &sri).unwrap(); assert_eq!(buf, b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_open() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut handle = crate::ToLinker::open(&dir, "my-key", target).await.unwrap(); let mut buf = Vec::new(); handle.read_to_end(&mut buf).await.unwrap(); handle.commit().await.unwrap(); assert_eq!(buf, b"hello world"); let buf = crate::read_sync(&dir, "my-key").unwrap(); assert_eq!(buf, b"hello world"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_open_hash() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut handle = crate::ToLinker::open_hash(&dir, target).await.unwrap(); let mut buf = Vec::new(); handle.read_to_end(&mut buf).await.unwrap(); let sri = handle.commit().await.unwrap(); assert_eq!(buf, b"hello world"); let buf = crate::read_hash_sync(&dir, &sri).unwrap(); assert_eq!(buf, b"hello world"); } #[test] fn test_open_sync() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut handle = crate::SyncToLinker::open(&dir, "my-key", target).unwrap(); let mut buf = Vec::new(); handle.read_to_end(&mut buf).unwrap(); handle.commit().unwrap(); assert_eq!(buf, b"hello world"); let buf = crate::read_sync(&dir, "my-key").unwrap(); assert_eq!(buf, b"hello world"); } #[test] fn test_open_hash_sync() { let tmp = tempfile::tempdir().unwrap(); let target = create_tmpfile(&tmp, b"hello world"); let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let mut handle = crate::SyncToLinker::open_hash(&dir, target).unwrap(); let mut buf = Vec::new(); handle.read_to_end(&mut buf).unwrap(); let sri = handle.commit().unwrap(); assert_eq!(buf, b"hello world"); let buf = crate::read_hash_sync(&dir, &sri).unwrap(); assert_eq!(buf, b"hello world"); } } cacache-13.0.0/src/ls.rs000064400000000000000000000013611046102023000130300ustar 00000000000000//! Functions for iterating over the cache. use std::path::Path; use crate::errors::Result; use crate::index; /// Returns a synchronous iterator that lists all cache index entries. pub fn list_sync>(cache: P) -> impl Iterator> { index::ls(cache.as_ref()) } #[cfg(test)] mod tests { use super::*; #[test] fn test_list_sync() { // check that the public interface to list elements can actually use the // Iterator::Item let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); assert!(list_sync(dir) .map(|x| Ok(x?.key)) .collect::>>() .is_err()) } } cacache-13.0.0/src/put.rs000064400000000000000000000514501046102023000132260ustar 00000000000000//! Functions for writing to cache. use std::io::prelude::*; use std::path::{Path, PathBuf}; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::pin::Pin; use serde_json::Value; use ssri::{Algorithm, Integrity}; #[cfg(any(feature = "async-std", feature = "tokio"))] use crate::async_lib::{AsyncWrite, AsyncWriteExt}; use crate::content::write; use crate::errors::{Error, IoErrorExt, Result}; use crate::index; #[cfg(any(feature = "async-std", feature = "tokio"))] use std::task::{Context as TaskContext, Poll}; /// Writes `data` to the `cache`, indexing it under `key`. /// /// ## Example /// ```no_run /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::write("./my-cache", "my-key", b"hello").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn write(cache: P, key: K, data: D) -> Result where P: AsRef, D: AsRef<[u8]>, K: AsRef, { write_with_algo(Algorithm::Sha256, cache, key, data).await } /// Writes `data` to the `cache`, indexing it under `key`. Use this function /// to customize the hashing algorithm. /// /// ## Example /// ```no_run /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::write_with_algo(cacache::Algorithm::Xxh3, "./my-cache", "my-key", b"hello").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn write_with_algo( algo: Algorithm, cache: P, key: K, data: D, ) -> Result where P: AsRef, D: AsRef<[u8]>, K: AsRef, { async fn inner(algo: Algorithm, cache: &Path, key: &str, data: &[u8]) -> Result { let mut writer = WriteOpts::new() .algorithm(algo) .size(data.len()) .open(cache, key) .await?; writer.write_all(data).await.with_context(|| { format!("Failed to write to cache data for key {key} for cache at {cache:?}") })?; writer.commit().await } inner(algo, cache.as_ref(), key.as_ref(), data.as_ref()).await } /// Writes `data` to the `cache`, skipping associating an index key with it. /// /// ## Example /// ```no_run /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::write_hash("./my-cache", b"hello").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn write_hash(cache: P, data: D) -> Result where P: AsRef, D: AsRef<[u8]>, { write_hash_with_algo(Algorithm::Sha256, cache, data).await } /// Writes `data` to the `cache`, skipping associating an index key with it. /// Use this to customize the hashing algorithm. /// /// ## Example /// ```no_run /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// cacache::write_hash_with_algo(cacache::Algorithm::Xxh3, "./my-cache", b"hello").await?; /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn write_hash_with_algo(algo: Algorithm, cache: P, data: D) -> Result where P: AsRef, D: AsRef<[u8]>, { async fn inner(algo: Algorithm, cache: &Path, data: &[u8]) -> Result { let mut writer = WriteOpts::new() .algorithm(algo) .size(data.len()) .open_hash(cache) .await?; writer .write_all(data) .await .with_context(|| format!("Failed to write to cache data for cache at {cache:?}"))?; writer.commit().await } inner(algo, cache.as_ref(), data.as_ref()).await } /// A reference to an open file writing to the cache. #[cfg(any(feature = "async-std", feature = "tokio"))] pub struct Writer { cache: PathBuf, key: Option, written: usize, pub(crate) writer: write::AsyncWriter, opts: WriteOpts, } #[cfg(any(feature = "async-std", feature = "tokio"))] impl AsyncWrite for Writer { fn poll_write( mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>, buf: &[u8], ) -> Poll> { let amt = futures::ready!(Pin::new(&mut self.writer).poll_write(cx, buf))?; self.written += amt; Poll::Ready(Ok(amt)) } fn poll_flush(mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>) -> Poll> { Pin::new(&mut self.writer).poll_flush(cx) } #[cfg(feature = "async-std")] fn poll_close(mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>) -> Poll> { Pin::new(&mut self.writer).poll_close(cx) } #[cfg(feature = "tokio")] fn poll_shutdown( mut self: Pin<&mut Self>, cx: &mut TaskContext<'_>, ) -> Poll> { Pin::new(&mut self.writer).poll_shutdown(cx) } } #[cfg(any(feature = "async-std", feature = "tokio"))] impl Writer { /// Creates a new writable file handle into the cache. /// /// ## Example /// ```no_run /// use async_attributes; /// use async_std::prelude::*; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let mut fd = cacache::Writer::create("./my-cache", "my-key").await?; /// fd.write_all(b"hello world").await.expect("Failed to write to cache"); /// // Data is not saved into the cache until you commit it. /// fd.commit().await?; /// Ok(()) /// } /// ``` pub async fn create(cache: P, key: K) -> Result where P: AsRef, K: AsRef, { Self::create_with_algo(Algorithm::Sha256, cache, key).await } /// Creates a new writable file handle into the cache. Use this to /// customize the algorithm used for hashing. /// /// ## Example /// ```no_run /// use async_attributes; /// use async_std::prelude::*; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let mut fd = cacache::Writer::create_with_algo(cacache::Algorithm::Xxh3, "./my-cache", "my-key").await?; /// fd.write_all(b"hello world").await.expect("Failed to write to cache"); /// // Data is not saved into the cache until you commit it. /// fd.commit().await?; /// Ok(()) /// } /// ``` pub async fn create_with_algo(algo: Algorithm, cache: P, key: K) -> Result where P: AsRef, K: AsRef, { async fn inner(algo: Algorithm, cache: &Path, key: &str) -> Result { WriteOpts::new().algorithm(algo).open(cache, key).await } inner(algo, cache.as_ref(), key.as_ref()).await } /// Closes the Writer handle and writes content and index entries. Also /// verifies data against `size` and `integrity` options, if provided. /// Must be called manually in order to complete the writing process, /// otherwise everything will be thrown out. pub async fn commit(mut self) -> Result { let cache = self.cache; let writer_sri = self.writer.close().await?; if let Some(sri) = &self.opts.sri { if sri.matches(&writer_sri).is_none() { return Err(ssri::Error::IntegrityCheckError(sri.clone(), writer_sri).into()); } } else { self.opts.sri = Some(writer_sri.clone()); } if let Some(size) = self.opts.size { if size != self.written { return Err(Error::SizeMismatch(size, self.written)); } } if let Some(key) = self.key { index::insert_async(&cache, &key, self.opts).await } else { Ok(writer_sri) } } } /// Writes `data` to the `cache` synchronously, indexing it under `key`. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let data = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// Ok(()) /// } /// ``` pub fn write_sync(cache: P, key: K, data: D) -> Result where P: AsRef, D: AsRef<[u8]>, K: AsRef, { write_sync_with_algo(Algorithm::Sha256, cache, key, data) } /// Writes `data` to the `cache` synchronously, indexing it under `key`. Use /// this to customize the hashing algorithm. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let data = cacache::write_sync_with_algo(cacache::Algorithm::Xxh3, "./my-cache", "my-key", b"hello")?; /// Ok(()) /// } /// ``` pub fn write_sync_with_algo( algo: Algorithm, cache: P, key: K, data: D, ) -> Result where P: AsRef, D: AsRef<[u8]>, K: AsRef, { fn inner(algo: Algorithm, cache: &Path, key: &str, data: &[u8]) -> Result { let mut writer = SyncWriter::create_with_algo(algo, cache, key)?; writer.write_all(data).with_context(|| { format!("Failed to write to cache data for key {key} for cache at {cache:?}") })?; writer.written = data.as_ref().len(); writer.commit() } inner(algo, cache.as_ref(), key.as_ref(), data.as_ref()) } /// Writes `data` to the `cache` synchronously, skipping associating a key with it. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let data = cacache::write_hash_sync("./my-cache", b"hello")?; /// Ok(()) /// } /// ``` pub fn write_hash_sync(cache: P, data: D) -> Result where P: AsRef, D: AsRef<[u8]>, { write_hash_sync_with_algo(Algorithm::Sha256, cache, data) } /// Writes `data` to the `cache` synchronously, skipping associating a key with it. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let data = cacache::write_hash_sync_with_algo(cacache::Algorithm::Xxh3, "./my-cache", b"hello")?; /// Ok(()) /// } /// ``` pub fn write_hash_sync_with_algo(algo: Algorithm, cache: P, data: D) -> Result where P: AsRef, D: AsRef<[u8]>, { fn inner(algo: Algorithm, cache: &Path, data: &[u8]) -> Result { let mut writer = WriteOpts::new() .algorithm(algo) .size(data.len()) .open_hash_sync(cache)?; writer .write_all(data) .with_context(|| format!("Failed to write to cache data for cache at {cache:?}"))?; writer.written = data.len(); writer.commit() } inner(algo, cache.as_ref(), data.as_ref()) } /// Builder for options and flags for opening a new cache file to write data into. #[derive(Clone, Default)] pub struct WriteOpts { pub(crate) algorithm: Option, pub(crate) sri: Option, pub(crate) size: Option, pub(crate) time: Option, pub(crate) metadata: Option, pub(crate) raw_metadata: Option>, } impl WriteOpts { /// Creates a blank set of cache writing options. pub fn new() -> WriteOpts { Default::default() } /// Opens the file handle for writing, returning an Writer instance. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn open(self, cache: P, key: K) -> Result where P: AsRef, K: AsRef, { async fn inner(me: WriteOpts, cache: &Path, key: &str) -> Result { Ok(Writer { cache: cache.to_path_buf(), key: Some(String::from(key)), written: 0, writer: write::AsyncWriter::new( cache, me.algorithm.unwrap_or(Algorithm::Sha256), None, ) .await?, opts: me, }) } inner(self, cache.as_ref(), key.as_ref()).await } /// Opens the file handle for writing, without a key returning an Writer instance. #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn open_hash

(self, cache: P) -> Result where P: AsRef, { async fn inner(me: WriteOpts, cache: &Path) -> Result { Ok(Writer { cache: cache.to_path_buf(), key: None, written: 0, writer: write::AsyncWriter::new( cache, me.algorithm.unwrap_or(Algorithm::Sha256), me.size, ) .await?, opts: me, }) } inner(self, cache.as_ref()).await } /// Opens the file handle for writing synchronously, returning a SyncWriter instance. pub fn open_sync(self, cache: P, key: K) -> Result where P: AsRef, K: AsRef, { fn inner(me: WriteOpts, cache: &Path, key: &str) -> Result { Ok(SyncWriter { cache: cache.to_path_buf(), key: Some(String::from(key)), written: 0, writer: write::Writer::new( cache, me.algorithm.unwrap_or(Algorithm::Sha256), me.size, )?, opts: me, }) } inner(self, cache.as_ref(), key.as_ref()) } /// Opens the file handle for writing, without a key returning an SyncWriter instance. pub fn open_hash_sync

(self, cache: P) -> Result where P: AsRef, { fn inner(me: WriteOpts, cache: &Path) -> Result { Ok(SyncWriter { cache: cache.to_path_buf(), key: None, written: 0, writer: write::Writer::new( cache, me.algorithm.unwrap_or(Algorithm::Sha256), me.size, )?, opts: me, }) } inner(self, cache.as_ref()) } /// Configures the algorithm to write data under. pub fn algorithm(mut self, algo: Algorithm) -> Self { self.algorithm = Some(algo); self } /// Sets the expected size of the data to write. If there's a date size /// mismatch, `put.commit()` will return an error. pub fn size(mut self, size: usize) -> Self { self.size = Some(size); self } /// Sets arbitrary additional metadata to associate with the index entry. pub fn metadata(mut self, metadata: Value) -> Self { self.metadata = Some(metadata); self } /// Sets arbitrary additional binary metadata to associate with the index entry. pub fn raw_metadata(mut self, metadata: Vec) -> Self { self.raw_metadata = Some(metadata); self } /// Sets the specific time in unix milliseconds to associate with this /// entry. This is usually automatically set to the write time, but can be /// useful to change for tests and such. pub fn time(mut self, time: u128) -> Self { self.time = Some(time); self } /// Sets the expected integrity hash of the written data. If there's a /// mismatch between this Integrity and the one calculated by the write, /// `put.commit()` will error. pub fn integrity(mut self, sri: Integrity) -> Self { self.sri = Some(sri); self } } /// A reference to an open file writing to the cache. pub struct SyncWriter { cache: PathBuf, key: Option, written: usize, pub(crate) writer: write::Writer, opts: WriteOpts, } impl Write for SyncWriter { fn write(&mut self, buf: &[u8]) -> std::io::Result { let written = self.writer.write(buf)?; self.written += written; Ok(written) } fn flush(&mut self) -> std::io::Result<()> { self.writer.flush() } } impl SyncWriter { /// Creates a new writable file handle into the cache. /// /// ## Example /// ```no_run /// use std::io::prelude::*; /// /// fn main() -> cacache::Result<()> { /// let mut fd = cacache::SyncWriter::create("./my-cache", "my-key")?; /// fd.write_all(b"hello world").expect("Failed to write to cache"); /// // Data is not saved into the cache until you commit it. /// fd.commit()?; /// Ok(()) /// } /// ``` pub fn create(cache: P, key: K) -> Result where P: AsRef, K: AsRef, { fn inner(cache: &Path, key: &str) -> Result { WriteOpts::new() .algorithm(Algorithm::Sha256) .open_sync(cache, key) } inner(cache.as_ref(), key.as_ref()) } /// Creates a new writable file handle into the cache. Use this to /// customize the hashing algorithm. /// /// ## Example /// ```no_run /// use std::io::prelude::*; /// /// fn main() -> cacache::Result<()> { /// let mut fd = cacache::SyncWriter::create_with_algo(cacache::Algorithm::Xxh3, "./my-cache", "my-key")?; /// fd.write_all(b"hello world").expect("Failed to write to cache"); /// // Data is not saved into the cache until you commit it. /// fd.commit()?; /// Ok(()) /// } /// ``` pub fn create_with_algo(algo: Algorithm, cache: P, key: K) -> Result where P: AsRef, K: AsRef, { fn inner(algo: Algorithm, cache: &Path, key: &str) -> Result { WriteOpts::new().algorithm(algo).open_sync(cache, key) } inner(algo, cache.as_ref(), key.as_ref()) } /// Closes the Writer handle and writes content and index entries. Also /// verifies data against `size` and `integrity` options, if provided. /// Must be called manually in order to complete the writing process, /// otherwise everything will be thrown out. pub fn commit(mut self) -> Result { let cache = self.cache; let writer_sri = self.writer.close()?; if let Some(sri) = &self.opts.sri { if sri.matches(&writer_sri).is_none() { return Err(ssri::Error::IntegrityCheckError(sri.clone(), writer_sri).into()); } } else { self.opts.sri = Some(writer_sri.clone()); } if let Some(size) = self.opts.size { if size != self.written { return Err(Error::SizeMismatch(size, self.written)); } } if let Some(key) = self.key { index::insert(&cache, &key, self.opts) } else { Ok(writer_sri) } } } #[cfg(test)] mod tests { #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn round_trip() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::write(&dir, "hello", b"hello").await.unwrap(); let data = crate::read(&dir, "hello").await.unwrap(); assert_eq!(data, b"hello"); } #[test] fn round_trip_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); crate::write_sync(&dir, "hello", b"hello").unwrap(); let data = crate::read_sync(&dir, "hello").unwrap(); assert_eq!(data, b"hello"); } #[test] fn hash_write_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let original = format!("hello world{}", 5); let integrity = crate::write_hash_sync(&dir, &original) .expect("should be able to write a hash synchronously"); let bytes = crate::read_hash_sync(&dir, &integrity) .expect("should be able to read the data we just wrote"); let result = String::from_utf8(bytes).expect("we wrote valid utf8 but did not read valid utf8 back"); assert_eq!(result, original, "we did not read back what we wrote"); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn hash_write_async() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let original = format!("hello world{}", 12); let integrity = crate::write_hash(&dir, &original) .await .expect("should be able to write a hash asynchronously"); let bytes = crate::read_hash(&dir, &integrity) .await .expect("should be able to read back what we wrote"); let result = String::from_utf8(bytes).expect("we wrote valid utf8 but did not read valid utf8 back"); assert_eq!(result, original, "we did not read back what we wrote"); } } cacache-13.0.0/src/rm.rs000064400000000000000000000217001046102023000130270ustar 00000000000000//! Functions for removing things from the cache. use std::fs; use std::path::Path; use ssri::Integrity; use crate::content::rm; use crate::errors::{IoErrorExt, Result}; use crate::index; /// Removes an individual index metadata entry. The associated content will be /// left in the cache. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello").await?; /// /// cacache::remove("./my-cache", "my-key").await?; /// /// // This fails: /// cacache::read("./my-cache", "my-key").await?; /// /// // But this succeeds: /// cacache::read_hash("./my-cache", &sri).await?; /// /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn remove(cache: P, key: K) -> Result<()> where P: AsRef, K: AsRef, { index::delete_async(cache.as_ref(), key.as_ref()).await } /// Removes an individual content entry. Any index entries pointing to this /// content will become invalidated. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello").await?; /// /// cacache::remove_hash("./my-cache", &sri).await?; /// /// // These fail: /// cacache::read("./my-cache", "my-key").await?; /// cacache::read_hash("./my-cache", &sri).await?; /// /// // But this succeeds: /// cacache::metadata("./my-cache", "my-key").await?; /// /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn remove_hash>(cache: P, sri: &Integrity) -> Result<()> { rm::rm_async(cache.as_ref(), sri).await } /// Removes entire contents of the cache, including temporary files, the entry /// index, and all content data. /// /// ## Example /// ```no_run /// use async_std::prelude::*; /// use async_attributes; /// /// #[async_attributes::main] /// async fn main() -> cacache::Result<()> { /// let sri = cacache::write("./my-cache", "my-key", b"hello").await?; /// /// cacache::clear("./my-cache").await?; /// /// // These all fail: /// cacache::read("./my-cache", "my-key").await?; /// cacache::metadata("./my-cache", "my-key").await?; /// cacache::read_hash("./my-cache", &sri).await?; /// /// Ok(()) /// } /// ``` #[cfg(any(feature = "async-std", feature = "tokio"))] pub async fn clear>(cache: P) -> Result<()> { async fn inner(cache: &Path) -> Result<()> { for entry in cache .read_dir() .with_context(|| { format!( "Failed to read directory contents while clearing cache, at {}", cache.display() ) })? .flatten() { crate::async_lib::remove_dir_all(entry.path()) .await .with_context(|| format!("Failed to clear cache at {}", cache.display()))?; } Ok(()) } inner(cache.as_ref()).await } /// Removes an individual index entry synchronously. The associated content /// will be left in the cache. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// /// cacache::remove_sync("./my-cache", "my-key")?; /// /// // This fails: /// cacache::read_sync("./my-cache", "my-key")?; /// /// // But this succeeds: /// cacache::read_hash_sync("./my-cache", &sri)?; /// /// Ok(()) /// } /// ``` pub fn remove_sync(cache: P, key: K) -> Result<()> where P: AsRef, K: AsRef, { index::delete(cache.as_ref(), key.as_ref()) } /// Removes an individual content entry synchronously. Any index entries /// pointing to this content will become invalidated. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// /// cacache::remove_hash_sync("./my-cache", &sri)?; /// /// // These fail: /// cacache::read_sync("./my-cache", "my-key")?; /// cacache::read_hash_sync("./my-cache", &sri)?; /// /// // But this succeeds: /// cacache::metadata_sync("./my-cache", "my-key")?; /// /// Ok(()) /// } /// ``` pub fn remove_hash_sync>(cache: P, sri: &Integrity) -> Result<()> { rm::rm(cache.as_ref(), sri) } /// Removes entire contents of the cache synchronously, including temporary /// files, the entry index, and all content data. /// /// ## Example /// ```no_run /// use std::io::Read; /// /// fn main() -> cacache::Result<()> { /// let sri = cacache::write_sync("./my-cache", "my-key", b"hello")?; /// /// cacache::clear_sync("./my-cache")?; /// /// // These all fail: /// cacache::read_sync("./my-cache", "my-key")?; /// cacache::read_hash_sync("./my-cache", &sri)?; /// cacache::metadata_sync("./my-cache", "my-key")?; /// /// Ok(()) /// } /// ``` pub fn clear_sync>(cache: P) -> Result<()> { fn inner(cache: &Path) -> Result<()> { for entry in cache .read_dir() .with_context(|| { format!( "Failed to read directory contents while clearing cache, at {}", cache.display() ) })? .flatten() { fs::remove_dir_all(entry.path()) .with_context(|| format!("Failed to clear cache at {}", cache.display()))?; } Ok(()) } inner(cache.as_ref()) } #[cfg(test)] mod tests { #[cfg(feature = "async-std")] use async_attributes::test as async_test; #[cfg(feature = "tokio")] use tokio::test as async_test; #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_remove() { futures::executor::block_on(async { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write(&dir, "key", b"my-data").await.unwrap(); crate::remove(&dir, "key").await.unwrap(); let entry = crate::metadata(&dir, "key").await.unwrap(); assert_eq!(entry, None); let data_exists = crate::exists(&dir, &sri).await; assert!(data_exists); }); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_remove_data() { futures::executor::block_on(async { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write(&dir, "key", b"my-data").await.unwrap(); crate::remove_hash(&dir, &sri).await.unwrap(); let entry = crate::metadata(&dir, "key").await.unwrap(); assert!(entry.is_some()); let data_exists = crate::exists(&dir, &sri).await; assert!(!data_exists); }); } #[cfg(any(feature = "async-std", feature = "tokio"))] #[async_test] async fn test_clear() { futures::executor::block_on(async { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write(&dir, "key", b"my-data").await.unwrap(); crate::clear(&dir).await.unwrap(); let entry = crate::metadata(&dir, "key").await.unwrap(); assert!(entry.is_none()); let data_exists = crate::exists(&dir, &sri).await; assert!(!data_exists); }); } #[test] fn test_remove_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write_sync(&dir, "key", b"my-data").unwrap(); crate::remove_sync(&dir, "key").unwrap(); let new_entry = crate::metadata_sync(&dir, "key").unwrap(); assert!(new_entry.is_none()); let data_exists = crate::exists_sync(&dir, &sri); assert!(data_exists); } #[test] fn test_remove_data_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write_sync(&dir, "key", b"my-data").unwrap(); crate::remove_hash_sync(&dir, &sri).unwrap(); let entry = crate::metadata_sync(&dir, "key").unwrap(); assert!(entry.is_some()); let data_exists = crate::exists_sync(&dir, &sri); assert!(!data_exists); } #[test] fn test_clear_sync() { let tmp = tempfile::tempdir().unwrap(); let dir = tmp.path().to_owned(); let sri = crate::write_sync(&dir, "key", b"my-data").unwrap(); crate::clear_sync(&dir).unwrap(); let entry = crate::metadata_sync(&dir, "key").unwrap(); assert_eq!(entry, None); let data_exists = crate::exists_sync(&dir, &sri); assert!(!data_exists); } }