jwalk-0.8.1/.cargo_vcs_info.json0000644000000001360000000000100121520ustar { "git": { "sha1": "0079deb9faed6be48e77676494351f06411db5de" }, "path_in_vcs": "" }jwalk-0.8.1/.github/workflows/ci.yml000064400000000000000000000005461046102023000154620ustar 00000000000000name: Rust on: push: branches: [ main ] pull_request: branches: [ main ] jobs: build-and-test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - uses: Swatinem/rust-cache@v2 - name: tests run: cargo test - name: docs run: cargo doc - name: bench run: JWALK_BENCHMARK_DIR=~/ cargo bench jwalk-0.8.1/.gitignore000064400000000000000000000014741046102023000127400ustar 00000000000000*~ .#* .DS_Store .cproject .hg/ .hgignore .idea *.iml __pycache__/ *.py[cod] *$py.class .project .settings/ .valgrindrc .vscode/ .favorites.json /*-*-*-*/ /*-*-*/ /Makefile /build /config.toml /dist/ /dl/ /doc /inst/ /llvm/ /mingw-build/ /nd/ /obj/ /rt/ /rustllvm/ /src/libcore/unicode/DerivedCoreProperties.txt /src/libcore/unicode/DerivedNormalizationProps.txt /src/libcore/unicode/PropList.txt /src/libcore/unicode/ReadMe.txt /src/libcore/unicode/Scripts.txt /src/libcore/unicode/SpecialCasing.txt /src/libcore/unicode/UnicodeData.txt /stage[0-9]+/ /target target/ /test/ /tmp/ tags tags.* TAGS TAGS.* \#* \#*\# config.mk config.stamp keywords.md lexer.ml mir_dump Session.vim src/etc/dl tmp.*.rs version.md version.ml version.texi .cargo !src/vendor/** /src/target/ no_llvm_build /benches/assets/linux_checkout Cargo.lock jwalk-0.8.1/CHANGELOG.md000064400000000000000000000671621046102023000125670ustar 00000000000000All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## 0.8.1 (2022-12-15) ### New Features - re-export `rayon` in the crate root. This makes creating a `ThreadPool` easier as it doesn't force us to maintain our own `rayon` dependency. ### Commit Statistics - 1 commit contributed to the release. - 1 commit was understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - re-export `rayon` in the crate root. ([`b49e157`](https://github.com/Byron/jwalk/commit/b49e157b539150a44b761e43d8b09621367e760c))
## 0.8.0 (2022-12-15) ### New Features (BREAKING) - `Parallelism::RayonExistingPool::busy_timeout` is now optional. That way we can indicate that no waiting should be done as we know the given threadpool has enough resources. ### Commit Statistics - 2 commits contributed to the release. - 1 commit was understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - Release jwalk v0.8.0 ([`be0bd21`](https://github.com/Byron/jwalk/commit/be0bd21bd5213033ac55b90ec7753d6e72b4bd84)) - `Parallelism::RayonExistingPool::busy_timeout` is now optional. ([`3a71721`](https://github.com/Byron/jwalk/commit/3a717219411a7478b90c4d694d57e28d8941dde1))
## 0.7.0 (2022-12-15) This release makes iterator creation fallible to avoid potential hangs when there is no available thread to process any of the iterator work. ### New Features - `WalkDirGeneric::try_into_iter()` for early error handling. If we can't instantiate the iterator due to a busy thread-pool, we can now abort early instead of yielding a fake-entry just to show an error occurred. This is the preferred way to instantiate a `jwalk` iterator. ### New Features (BREAKING) - Detect possible deadlocks when instantiating a parallel iterator. Deadlocks can happen if the producer for results doesn't start as there is no free thread on the rayon pool, and the only way for it to become free is if the iterator produces results. We now offer a `busy_timeout` in the relevant variants of the `Parallelism` enumeration to allow controlling how long we will wait until we abort with an error. ### Commit Statistics - 7 commits contributed to the release. - 1 day passed between releases. - 2 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Thanks Clippy [Clippy](https://github.com/rust-lang/rust-clippy) helped 1 time to make code idiomatic. ### Commit Details
view details * **Uncategorized** - Release jwalk v0.7.0 ([`c265744`](https://github.com/Byron/jwalk/commit/c265744d74b3ea231eacee6332a99ee292f3018a)) - prepare changelog prior to release ([`67364f9`](https://github.com/Byron/jwalk/commit/67364f910f1ba1ffeb5c30d0f75709431b212e2b)) - refactor ([`a94d14b`](https://github.com/Byron/jwalk/commit/a94d14b34980e5dd53ea6dde9c5676f44c80a7fa)) - thanks clippy ([`7e300c6`](https://github.com/Byron/jwalk/commit/7e300c68691f462ebb0848db915d7798b49dfccc)) - `WalkDirGeneric::try_into_iter()` for early error handling. ([`7d5b8b8`](https://github.com/Byron/jwalk/commit/7d5b8b870bfca2b1b68de1427fbdc0ec1a1bff2b)) - Detect possible deadlocks when instantiating a parallel iterator. ([`3bf1bc2`](https://github.com/Byron/jwalk/commit/3bf1bc226571869e4a5c357d4f6e40ad0a28f3ff)) - fix various IDE warnings ([`cc0009f`](https://github.com/Byron/jwalk/commit/cc0009f626ac86f92e20ce3d4e7c2a8f00d979a0))
## 0.6.2 (2022-12-13) ### Bug Fixes - stalling issue when threadpool is used is no more. The issue seems to have been that `install` blocks whereas `spawn` properly releases the main thread. This seems to have been changed subtly due to changes in `rayon`, which breaks an assumption on how the code is executed. Replacing `install()` with `spawn` calls resolved the issue. ### Commit Statistics - 5 commits contributed to the release. - 1 commit was understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - Release jwalk v0.6.2 ([`2d1b2fb`](https://github.com/Byron/jwalk/commit/2d1b2fbe59a0ebb0413b54a8b8b0bba713e4d0e3)) - Merge branch 'stalling-issue' ([`7bd2f35`](https://github.com/Byron/jwalk/commit/7bd2f35d4fd106edbafa187ef4481333bb60da7d)) - stalling issue when threadpool is used is no more. ([`bd3e880`](https://github.com/Byron/jwalk/commit/bd3e88017ea29c3b89b518f3a721ba35577b7666)) - refactor ([`1032308`](https://github.com/Byron/jwalk/commit/10323089dbf00e01a0280a35f826ca269b6eeea6)) - print each path seen during iteration ([`5e83ad5`](https://github.com/Byron/jwalk/commit/5e83ad5f09852a6449f63b0c954eec81413de1c2))
## 0.6.1 (2022-12-13) The first release under new ownership with no user-facing changes. - The project uses GitHub CI and `cargo smart-release` for releases. - some code cleanup based on `cargo clippy`. ### Changing project ownership to @Byron - Thanks and good luck! (By https://github.com/jessegrosjean) - Thank you, my pleasure (By https://github.com/Byron) ### Commit Statistics - 30 commits contributed to the release over the course of 705 calendar days. - 705 days passed between releases. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Thanks Clippy [Clippy](https://github.com/rust-lang/rust-clippy) helped 1 time to make code idiomatic. ### Commit Details
view details * **Uncategorized** - Release jwalk v0.6.1 ([`6a2781c`](https://github.com/Byron/jwalk/commit/6a2781c6211a6db777c08bcdeb60f0317c00bc3e)) - prepare changelog prior to release ([`c772967`](https://github.com/Byron/jwalk/commit/c77296707df392193dc47bcd1f465fa813215b82)) - another round of link adjustments ([`7c12dc3`](https://github.com/Byron/jwalk/commit/7c12dc333d5086ed41228ee410653def9ff5adf7)) - thanks clippy ([`51e2b0d`](https://github.com/Byron/jwalk/commit/51e2b0d0330b264972422d44ca25affcced981d5)) - run benchmarks on CI ([`cc0fd74`](https://github.com/Byron/jwalk/commit/cc0fd74d439fb24b063d3d6d4d05f3370d41bf65)) - set version back to what's current, change URLs and add myself as author ([`aa5b24d`](https://github.com/Byron/jwalk/commit/aa5b24dff23b4fd3b48e6a3e4c59a504a380beda)) - cleanup tests ([`d8f0756`](https://github.com/Byron/jwalk/commit/d8f07566eb2487f6f73bd06edc8286d0bf3ee015)) - rename 'master' to 'main' ([`362d03c`](https://github.com/Byron/jwalk/commit/362d03c1059658f88a77b7b9db7d14a2ccc2e6b5)) - enable CI ([`a115e7a`](https://github.com/Byron/jwalk/commit/a115e7acbf6c147c2bfeba3981c6bc5d587c2aef)) - Moved to example/crash ([`1a09da5`](https://github.com/Byron/jwalk/commit/1a09da59cda994758a856c30f931afe4ee0208d8)) - Changing project ownership ([`cd5d1ae`](https://github.com/Byron/jwalk/commit/cd5d1aed268ad5a0692f698a419bdea835aa71a5)) - Add crash example ([`c0b262b`](https://github.com/Byron/jwalk/commit/c0b262bc4b4bbb0134c7987cbb6995a46a005070)) - More unneeded code removal ([`69c00ec`](https://github.com/Byron/jwalk/commit/69c00ec6f60b0010c4a99db7ea67093f401d8260)) - Added failing combine with rayon test ([`5eccf5e`](https://github.com/Byron/jwalk/commit/5eccf5efe09c1a24e839157b6098a9bc1c8948cc)) - Remove some unneeded rayon calls ([`f61a535`](https://github.com/Byron/jwalk/commit/f61a53585514f50f772a4cb81c57150ac1fd4450)) - Update to rayon 1.6.1, remove jwalk_par_bridge ([`94c0385`](https://github.com/Byron/jwalk/commit/94c0385959c4004aff3ec2c60e729fa654579141)) - Merge pull request #33 from bootandy/patch-2 ([`9575132`](https://github.com/Byron/jwalk/commit/9575132b76513bea64405a7cc5a98708d31d5743)) - Fix typo ([`9ae51a5`](https://github.com/Byron/jwalk/commit/9ae51a5823a2c530871b2ba7fbce5096a8d37339)) - Merge pull request #32 from Byron/master ([`5c857d4`](https://github.com/Byron/jwalk/commit/5c857d4e7fa2d587617e442d7a81e070c2c55175)) - Don't ignore hidden files in `du` example ([`0786beb`](https://github.com/Byron/jwalk/commit/0786bebf3962e862c56577da389d9b14dfb3b5f1)) - Remove unused imports in example ([`80a6d2e`](https://github.com/Byron/jwalk/commit/80a6d2e3054e84b36ae6c45791b5b62d579dbea7)) - Update readme ([`6f9ebf5`](https://github.com/Byron/jwalk/commit/6f9ebf54dcfcc561c1e0afe0b797fcef8dc65b51)) - Update to from latest rayon src/iter/par_bridge.rs ([`e3a46c1`](https://github.com/Byron/jwalk/commit/e3a46c1b02111725b7d3c7929a6b34079692f154)) - Add simple du example ([`ae905c6`](https://github.com/Byron/jwalk/commit/ae905c60762c72aab9230717c8ae7fd6e9fcf720)) - Rename new bench to "rayon" ([`5ee29f5`](https://github.com/Byron/jwalk/commit/5ee29f5755f7ca21c2f7d2d07b54d195c08acc72)) - Merge pull request #31 from Byron/master ([`4817c1a`](https://github.com/Byron/jwalk/commit/4817c1a6817f659f175bce271701bfbbad5b233b)) - reduce the recursion to its core, keep no state ([`247cf38`](https://github.com/Byron/jwalk/commit/247cf387210231ae699c5a247a024fb7724846ec)) - Support for file metadata to approximate typical tool usage ([`a6afcfe`](https://github.com/Byron/jwalk/commit/a6afcfe535eee18fc9ce216a6facee35b58d6ed1)) - Add simple recursive way of building a file tree for reference ([`9c66ef6`](https://github.com/Byron/jwalk/commit/9c66ef6009c47283b43b44cf08eefbdc25765737)) - (cargo-release) start next development iteration 0.6.0 ([`a1d5209`](https://github.com/Byron/jwalk/commit/a1d5209c32723e81ed7c76e0c0ee8e3de2a07748))
## v0.6.0 (2021-01-06) Added depth and path being read to params to ProcessReadDirFunction callback. Allow setting initial root_read_dir_state (ReadDirState) instead of always getting ::default() value. ### Commit Statistics - 2 commits contributed to the release over the course of 9 calendar days. - 9 days passed between releases. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - Change release to 0.6 because of breaking changes ([`d74cc13`](https://github.com/Byron/jwalk/commit/d74cc130c506d5b2744bb9cfda8078ac2da0208f)) - (cargo-release) start next development iteration 0.5.2 ([`6e4ba03`](https://github.com/Byron/jwalk/commit/6e4ba039756cccff449a317576705c4979bb8fbc))
## v0.5.2 (2020-12-28) ### Commit Statistics - 12 commits contributed to the release over the course of 289 calendar days. - 289 days passed between releases. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - (cargo-release) version 0.5.2 ([`85ef009`](https://github.com/Byron/jwalk/commit/85ef009dfe12ee23c1dacda54955281a441ec97a)) - update dependencies ([`ea66ef8`](https://github.com/Byron/jwalk/commit/ea66ef8b8a957566d994d59048be285a23f29462)) - Add more capability to ProcessReadDirFunction ([`2459776`](https://github.com/Byron/jwalk/commit/24597762a6bec16dcc5f421bdd0f3dee960f68cd)) - Add test processing jwalk entries with rayon par_bridge() ([`57860ff`](https://github.com/Byron/jwalk/commit/57860ff3cfd69dc172d35666d4986055b5ba2e05)) - cargo fmt ([`a431561`](https://github.com/Byron/jwalk/commit/a431561f11d59defcf0eed3707b46683d1f89655)) - Merge pull request #25 from brmmm3/fix_warnings ([`fe12f26`](https://github.com/Byron/jwalk/commit/fe12f2666db66cc3e056267438b7b7b9f946f3b7)) - Fix warnings ([`64ddb05`](https://github.com/Byron/jwalk/commit/64ddb05cbc130b865cc872d1e56c4ef71ad1a9bd)) - Merge branch 'master' of https://github.com/jessegrosjean/walk ([`32e46c6`](https://github.com/Byron/jwalk/commit/32e46c68fb3933a2fd6fc9a329b7a72305f5eb64)) - Note preload_metadata removal ([`3bd6618`](https://github.com/Byron/jwalk/commit/3bd6618d2a2dc689837abf97c1c9fc1f86475164)) - Merge pull request #23 from bootandy/patch-1 ([`b4776b9`](https://github.com/Byron/jwalk/commit/b4776b9c4d1ce6f194f542d92a795653c7960408)) - fix typo ([`d630f80`](https://github.com/Byron/jwalk/commit/d630f80335edcec2abdd6b0fb6a525565b9adda9)) - (cargo-release) start next development iteration 0.5.1 ([`6d93359`](https://github.com/Byron/jwalk/commit/6d93359836bd7c4e1ddefa4462eb8083f45a82b1))
## v0.5.1 (2020-03-13) ### Commit Statistics - 5 commits contributed to the release. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - (cargo-release) version 0.5.1 ([`1621bee`](https://github.com/Byron/jwalk/commit/1621bee2077484836558f41ec6e90be40643dcd6)) - More tests for relative paths ([`246c997`](https://github.com/Byron/jwalk/commit/246c99701e03a671274b2a17cd3660f2388fc9c4)) - Use path for root dir_entry.file_name if path has now filename of own ([`80f6f7a`](https://github.com/Byron/jwalk/commit/80f6f7ae7699651ec4684510eb70e4289e0ac28c)) - Merge pull request #19 from brmmm3/simplify_and_then ([`ff234bb`](https://github.com/Byron/jwalk/commit/ff234bbbcf8999585e6c3d7e4ba8c2b44f1b8d17)) - (cargo-release) start next development iteration 0.5.0 ([`5e97e11`](https://github.com/Byron/jwalk/commit/5e97e1159843b3ce39617a0a3ca2fea3e535e629))
## v0.5.0 (2020-03-13) First major change is that API and behavior are now closer to [`walkdir`] and jwalk now runs the majority of `walkdir`s tests. Second major change is the walk can now be parameterized with a client state type. This state can be manipulated from the `process_read_dir` callback and then is passed down when reading descendens with the `process_read_dir` callback. Part of this second change is that `preload_metadata` option is removed. That means `DirEntry.metadata()` is never a cached value. Instead you want to read metadata you should do it in the `process_entries` callback and store whatever values you need as `client_state`. See this [benchmark] as an example. ### Chore - Update criterion to 0.3 ### Commit Statistics - 26 commits contributed to the release over the course of 294 calendar days. - 294 days passed between releases. - 1 commit was understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - (cargo-release) version 0.5.0 ([`cf9d248`](https://github.com/Byron/jwalk/commit/cf9d248d13932b4afa531f704f8f3eb8e2d55ec3)) - Merge pull request #21 from jessegrosjean/symlinks ([`6410407`](https://github.com/Byron/jwalk/commit/641040798c71ab68fdba8874b6221f8e9d6295d7)) - Update dependencies ([`85dff6e`](https://github.com/Byron/jwalk/commit/85dff6e02f5c52b887b429d12411f9faf1db2a5c)) - Merge branch 'master' of https://github.com/jessegrosjean/walk into symlinks ([`0dfefa9`](https://github.com/Byron/jwalk/commit/0dfefa9e0e068a6f1902ffe0da5c99511bb88792)) - Get follow links working and passing tests ([`797f76f`](https://github.com/Byron/jwalk/commit/797f76f8c2cfcaf8b02c0891794d9310e2bf30f6)) - Clean ([`ff8f491`](https://github.com/Byron/jwalk/commit/ff8f491a1945ef2a148adffe9bb999b49ca2dc73)) - Merge pull request #18 from brmmm3/remove_unnecessary_clone ([`898ff91`](https://github.com/Byron/jwalk/commit/898ff914ae0c9d6f1f49e2fbe64b21742a93b524)) - Simplify and_then ([`b330914`](https://github.com/Byron/jwalk/commit/b3309145eee88c8b48daf1453d68e5d6d98d3ce7)) - Remove unnecessary clone. ([`0f8b1c8`](https://github.com/Byron/jwalk/commit/0f8b1c8350c4d31e6de88fddbc7fd61ecd990065)) - Merge pull request #12 from ignatenkobrain/patch-1 ([`f9a144b`](https://github.com/Byron/jwalk/commit/f9a144bd504027d44d0fd86c1aeef077ce3e543e)) - Update criterion to 0.3 ([`11fa0bc`](https://github.com/Byron/jwalk/commit/11fa0bc9e8541af333aafb41cc89218435474df4)) - Make Parallelism param actually have effect. Fix some related bugs. ([`3b90d8e`](https://github.com/Byron/jwalk/commit/3b90d8e40ba39d378077bb2f20da789af6addda0)) - use walkdir error struct ([`850954f`](https://github.com/Byron/jwalk/commit/850954f6b3756bbca66e2bccfde9f5f9297f2bb3)) - test/benches back to working ([`45529ef`](https://github.com/Byron/jwalk/commit/45529efd5496a4ab1694aeb5a82d8eab0f683e36)) - in progress more closesly follow walkdir ([`66d46ac`](https://github.com/Byron/jwalk/commit/66d46acfb2a375e30d1a904e1ab17a9b5855f31e)) - symlink work in progress ([`7ee0cb3`](https://github.com/Byron/jwalk/commit/7ee0cb3e72eb074f9081b5fd3cbc475e61fb9a43)) - Merge branch 'master' of https://github.com/jessegrosjean/walk ([`17ddb91`](https://github.com/Byron/jwalk/commit/17ddb917a12d3e9787b6cbb2ace28758a712a95e)) - Walk is now parameterized with client_state type ([`4e4218f`](https://github.com/Byron/jwalk/commit/4e4218f7dcd187ace10bd793515822976dd0259c)) - Merge pull request #8 from vks/cleanup ([`1074566`](https://github.com/Byron/jwalk/commit/10745666a4c6620337d8d1c1c099ddc2a90d02c8)) - Fix compiler warnings ([`064ee60`](https://github.com/Byron/jwalk/commit/064ee604277c94b38ea9768447a88070fcb641d8)) - Remove Cargo.lock ([`8683484`](https://github.com/Byron/jwalk/commit/86834849ce473fd2a910caf5d890c3c6067dcac8)) - fix table formatting ([`0ef72d1`](https://github.com/Byron/jwalk/commit/0ef72d1838e732771cafcd05ef04a70280628017)) - Update benchmarks ([`fd9af20`](https://github.com/Byron/jwalk/commit/fd9af20e6e523449f07f3810fd8eb2987812ec13)) - Merge pull request #5 from spacekookie/patch-1 ([`c10dbaa`](https://github.com/Byron/jwalk/commit/c10dbaaaf2130de162c163c86f6d23bb1ace506a)) - Reformatting benchmarks table ([`99678be`](https://github.com/Byron/jwalk/commit/99678be587f1da6b869fe5afdd95c3a14b7666ef)) - (cargo-release) start next development iteration 0.4.0 ([`29b6b1e`](https://github.com/Byron/jwalk/commit/29b6b1ecedc85a2baa01fddee3b6e75dbc230b54))
## v0.4.0 (2019-05-24) ### Commit Statistics - 3 commits contributed to the release over the course of 91 calendar days. - 91 days passed between releases. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - (cargo-release) version 0.4.0 ([`5d68189`](https://github.com/Byron/jwalk/commit/5d681896e37c7518c9c31690467412f49fa3a418)) - Added content spec error reporting for root DirEntry ([`9643ce0`](https://github.com/Byron/jwalk/commit/9643ce091daa437b7dce082a25664688bb1ada90)) - (cargo-release) start next development iteration 0.3.0 ([`956b1a5`](https://github.com/Byron/jwalk/commit/956b1a5d615a7dda34238626bc5097afc4be0d12))
## v0.3.0 (2019-02-21) ### Commit Statistics - 8 commits contributed to the release over the course of 9 calendar days. - 9 days passed between releases. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - (cargo-release) version 0.3.0 ([`09ebe45`](https://github.com/Byron/jwalk/commit/09ebe4508697551cf411664047e69220b8bd7ddb)) - Update dependencies ([`f47607d`](https://github.com/Byron/jwalk/commit/f47607d959e497b2577d5b7ceb1b911405ba07a1)) - Spelling ([`b8e9aea`](https://github.com/Byron/jwalk/commit/b8e9aea52d1e3e0f643b42cf17d80ac467f775bf)) - Fix bug when max_depth was set to 0 ([`29c035e`](https://github.com/Byron/jwalk/commit/29c035e70990e9a442bb4f79735f1bd406688c03)) - Revert "Simplify, stop tracking depth in read dir specs." ([`331a896`](https://github.com/Byron/jwalk/commit/331a8964286b121f5bfe55140d2f4d7f0d08e28f)) - Simplify, stop tracking depth in read dir specs. ([`c1bffdd`](https://github.com/Byron/jwalk/commit/c1bffdd2bc56ddec23bac58f93c8bc52d8150015)) - More badges! ([`e0d3e3e`](https://github.com/Byron/jwalk/commit/e0d3e3e67bcd2602433a84cb8e729f093cf38637)) - (cargo-release) start next development iteration 0.2.1 ([`133d168`](https://github.com/Byron/jwalk/commit/133d168a4586313a4e5eceb2ea70c348f387a5b4))
## v0.2.1 (2019-02-11) ### Commit Statistics - 4 commits contributed to the release. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - (cargo-release) version 0.2.1 ([`1dee5e1`](https://github.com/Byron/jwalk/commit/1dee5e12ec7f07d2d177e116d3eceac0ec5e2bbf)) - Fix usage documentation. Other documentation updates. ([`2631dbd`](https://github.com/Byron/jwalk/commit/2631dbd01c92eaa50ca877c819151ef129fcb6ef)) - Update usage ([`04f897f`](https://github.com/Byron/jwalk/commit/04f897f2be26b576c06af2ecb327ecb4c8ddfcda)) - (cargo-release) start next development iteration 0.2.0 ([`da74631`](https://github.com/Byron/jwalk/commit/da746315a12af5a9bbe4ece4897303f938d7d469))
## v0.2.0 (2019-02-11) ### Commit Statistics - 37 commits contributed to the release over the course of 13 calendar days. - 0 commits were understood as [conventional](https://www.conventionalcommits.org). - 0 issues like '(#ID)' were seen in commit messages ### Commit Details
view details * **Uncategorized** - Expose DirEntry fields for easy destructure. ([`ad9404e`](https://github.com/Byron/jwalk/commit/ad9404e19e78b2714b2db3da4ab85a91a8dbc481)) - Clippy suggestions and more doc updates ([`8fd5ae1`](https://github.com/Byron/jwalk/commit/8fd5ae172783d0e2ce3a696540e12d2576e4db9c)) - Rename "children" to content. Simplify sort to boolean. ([`f432c4f`](https://github.com/Byron/jwalk/commit/f432c4f806b01cfcaa16f4573a1d89d6817ab0ca)) - More docs for ReadDirSpec ([`347bc92`](https://github.com/Byron/jwalk/commit/347bc92cec8181b6a25cc80a663345cd3d87a3bb)) - More docs on how DirEntry is implemented ([`0bb13ee`](https://github.com/Byron/jwalk/commit/0bb13eecfa2f3f5b6448b5a7c4701add75d857cf)) - Fixing too many keywords! ([`ff5fba1`](https://github.com/Byron/jwalk/commit/ff5fba1ebe53d0240d64fb6f5e16e0dc1a11d2f8)) - More readme tweaks ([`0b23848`](https://github.com/Byron/jwalk/commit/0b238487c71a600dad31f05d57ca7f49d337d40d)) - Shorter readme page. ([`0ee3b3c`](https://github.com/Byron/jwalk/commit/0ee3b3c58af31b95099d79446dc67de88967115c)) - More doc updates ([`b3ac65c`](https://github.com/Byron/jwalk/commit/b3ac65cdca26fc1ca9e936d35256df517025fbca)) - Remove unused dependency ([`301e2a5`](https://github.com/Byron/jwalk/commit/301e2a5c053675dee50960dbea319ebcea901f28)) - Add badge ([`ab07f7a`](https://github.com/Byron/jwalk/commit/ab07f7ae7890a4fc18f509d9bea30f6a73e78f78)) - Create .travis.yml ([`66828d7`](https://github.com/Byron/jwalk/commit/66828d74f5cb3119bda9d99a7993ad8373bb5166)) - Add usage ([`58c7707`](https://github.com/Byron/jwalk/commit/58c770758cfd9f2076893b463490ef2e94dd3659)) - split code up into more files ([`566da8f`](https://github.com/Byron/jwalk/commit/566da8f0ac14f470dc181471e3c98a940ea96641)) - More docs cleanup ([`ade977d`](https://github.com/Byron/jwalk/commit/ade977dc29c59850b8ceb6f00089080c8e3cee18)) - Fix readme headings ([`738d9f6`](https://github.com/Byron/jwalk/commit/738d9f616005f22285985134258e49b800ff92b6)) - Fix README example ([`1f42a3e`](https://github.com/Byron/jwalk/commit/1f42a3eee18e602d65d420a04a8aace5cb87a20a)) - tests and cleanup ([`5fc2859`](https://github.com/Byron/jwalk/commit/5fc2859b730992aeb8cd74ae7398393ea61e2ff6)) - Fix DirEntry depth ([`5cc9db7`](https://github.com/Byron/jwalk/commit/5cc9db710664d51948f29fbad42e4a1be19b7390)) - Much cleaner, ready for real testing now. ([`2e58da7`](https://github.com/Byron/jwalk/commit/2e58da7524859f571b0206d480a471b17c7e7e75)) - Preping to box instead of template client function. ([`5cd8e04`](https://github.com/Byron/jwalk/commit/5cd8e04b8cd0cf2bdd8ce8a329a081316b5d06af)) - Add more walk options and tests ([`c6f8385`](https://github.com/Byron/jwalk/commit/c6f8385f4d43a66d2c1b2a4bfc29fec1fcde42c1)) - Add hidden file for tests ([`6372a1a`](https://github.com/Byron/jwalk/commit/6372a1aebbcd49aead0833e18c874f0609d09a63)) - Add fts to bench ([`32a527d`](https://github.com/Byron/jwalk/commit/32a527d4e39d853234396289b7cb7e1d3c479232)) - Cleanup ([`d3899aa`](https://github.com/Byron/jwalk/commit/d3899aacebfde722a1a37b4ecbecb81ce701df8e)) - Always return ReadDirResutls, no longer an option ([`32d6b86`](https://github.com/Byron/jwalk/commit/32d6b865ea6b967bb8a7e66710c82039bcf25b80)) - Merge pull request #2 from jessegrosjean/ordered ([`5b4745e`](https://github.com/Byron/jwalk/commit/5b4745e47937fdcccc413f522dfea1a47125b483)) - Merge branch 'master' into ordered ([`3ecc5b6`](https://github.com/Byron/jwalk/commit/3ecc5b62bd86430f5ee4380f113288bdfc7df346)) - Ready for feedback? ([`5d9ee63`](https://github.com/Byron/jwalk/commit/5d9ee635e30697cf1beab1b224b918cbff214ec9)) - more work on work_tree ([`4ced894`](https://github.com/Byron/jwalk/commit/4ced894c71e82a96f4f3c644fe7ec1dd51ad124f)) - working on more generic "work tree" ([`ebf9789`](https://github.com/Byron/jwalk/commit/ebf9789a8799b5e1c9d12d52f303fcb6ca59e0b9)) - Merge pull request #1 from jessegrosjean/ordered ([`36b2d2f`](https://github.com/Byron/jwalk/commit/36b2d2fba8ff5b224f961b9bcbe486e196a90943)) - Cleaned up ([`6b0df4a`](https://github.com/Byron/jwalk/commit/6b0df4aa401541481da9a86d40bb574ef95b9ca6)) - parameterized walk ([`115a5fe`](https://github.com/Byron/jwalk/commit/115a5fefb83968e283ae48bd68c599547735fae6)) - In progress ordered walk ([`ba02dda`](https://github.com/Byron/jwalk/commit/ba02ddaa4cf48b3e2546ae0a2ca4859f8151df50)) - Adding ordered version ([`b63f810`](https://github.com/Byron/jwalk/commit/b63f810eb7ca3ad3b93ea4746ea99a8ce4b16ea6)) - init commit ([`8f64d5a`](https://github.com/Byron/jwalk/commit/8f64d5ae22a3221058f50c17bc97b3c90afac3e0))
jwalk-0.8.1/Cargo.lock0000644000000446510000000000100101370ustar # This file is automatically @generated by Cargo. # It is not intended for manual editing. version = 3 [[package]] name = "aho-corasick" version = "0.7.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cc936419f96fa211c1b9166887b38e5e40b19958e5b895be7c1f93adec7071ac" dependencies = [ "memchr", ] [[package]] name = "anes" version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299" [[package]] name = "atty" version = "0.2.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8" dependencies = [ "hermit-abi", "libc", "winapi", ] [[package]] name = "autocfg" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d468802bab17cbc0cc575e9b053f41e72aa36bfa6b7f55e3529ffa43161b97fa" [[package]] name = "bitflags" version = "1.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a" [[package]] name = "bstr" version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ba3569f383e8f1598449f1a423e72e99569137b47740b1da11ef19af3d5c3223" dependencies = [ "memchr", ] [[package]] name = "bumpalo" version = "3.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "572f695136211188308f16ad2ca5c851a712c464060ae6974944458eb83880ba" [[package]] name = "cast" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5" [[package]] name = "cfg-if" version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd" [[package]] name = "ciborium" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b0c137568cc60b904a7724001b35ce2630fd00d5d84805fbb608ab89509d788f" dependencies = [ "ciborium-io", "ciborium-ll", "serde", ] [[package]] name = "ciborium-io" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "346de753af073cc87b52b2083a506b38ac176a44cfb05497b622e27be899b369" [[package]] name = "ciborium-ll" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "213030a2b5a4e0c0892b6652260cf6ccac84827b83a85a534e178e3906c4cf1b" dependencies = [ "ciborium-io", "half", ] [[package]] name = "clap" version = "3.2.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "71655c45cb9845d3270c9d6df84ebe72b4dad3c2ba3f7023ad47c144e4e473a5" dependencies = [ "bitflags", "clap_lex", "indexmap", "textwrap", ] [[package]] name = "clap_lex" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2850f2f5a82cbf437dd5af4d49848fbdfc27c157c3d010345776f952765261c5" dependencies = [ "os_str_bytes", ] [[package]] name = "criterion" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e7c76e09c1aae2bc52b3d2f29e13c6572553b30c4aa1b8a49fd70de6412654cb" dependencies = [ "anes", "atty", "cast", "ciborium", "clap", "criterion-plot", "itertools", "lazy_static", "num-traits", "oorandom", "plotters", "rayon", "regex", "serde", "serde_derive", "serde_json", "tinytemplate", "walkdir", ] [[package]] name = "criterion-plot" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6b50826342786a51a89e2da3a28f1c32b06e387201bc2d19791f622c673706b1" dependencies = [ "cast", "itertools", ] [[package]] name = "crossbeam" version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2801af0d36612ae591caa9568261fddce32ce6e08a7275ea334a06a4ad021a2c" dependencies = [ "cfg-if", "crossbeam-channel", "crossbeam-deque", "crossbeam-epoch", "crossbeam-queue", "crossbeam-utils", ] [[package]] name = "crossbeam-channel" version = "0.5.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2dd04ddaf88237dc3b8d8f9a3c1004b506b54b3313403944054d23c0870c521" dependencies = [ "cfg-if", "crossbeam-utils", ] [[package]] name = "crossbeam-deque" version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "715e8152b692bba2d374b53d4875445368fdf21a94751410af607a5ac677d1fc" dependencies = [ "cfg-if", "crossbeam-epoch", "crossbeam-utils", ] [[package]] name = "crossbeam-epoch" version = "0.9.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "01a9af1f4c2ef74bb8aa1f7e19706bc72d03598c8a570bb5de72243c7a9d9d5a" dependencies = [ "autocfg", "cfg-if", "crossbeam-utils", "memoffset", "scopeguard", ] [[package]] name = "crossbeam-queue" version = "0.3.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d1cfb3ea8a53f37c40dea2c7bedcbd88bdfae54f5e2175d6ecaff1c988353add" dependencies = [ "cfg-if", "crossbeam-utils", ] [[package]] name = "crossbeam-utils" version = "0.8.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4fb766fa798726286dbbb842f174001dab8abc7b627a1dd86e0b7222a95d929f" dependencies = [ "cfg-if", ] [[package]] name = "either" version = "1.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "90e5c1c8368803113bf0c9584fc495a58b86dc8a29edbf8fe877d21d9507e797" [[package]] name = "fastrand" version = "1.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7a407cfaa3385c4ae6b23e84623d48c2798d06e3e6a1878f7f59f17b3f86499" dependencies = [ "instant", ] [[package]] name = "fnv" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" [[package]] name = "fs_extra" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2022715d62ab30faffd124d40b76f4134a550a87792276512b18d63272333394" [[package]] name = "globset" version = "0.4.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0a1e17342619edbc21a964c2afbeb6c820c6a2560032872f397bb97ea127bd0a" dependencies = [ "aho-corasick", "bstr", "fnv", "log", "regex", ] [[package]] name = "half" version = "1.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eabb4a44450da02c90444cf74558da904edde8fb4e9035a9a6a4e15445af0bd7" [[package]] name = "hashbrown" version = "0.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888" [[package]] name = "hermit-abi" version = "0.1.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "62b467343b94ba476dcb2500d242dadbb39557df889310ac77c5d99100aaac33" dependencies = [ "libc", ] [[package]] name = "ignore" version = "0.4.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "713f1b139373f96a2e0ce3ac931cd01ee973c3c5dd7c40c0c2efe96ad2b6751d" dependencies = [ "crossbeam-utils", "globset", "lazy_static", "log", "memchr", "regex", "same-file", "thread_local", "walkdir", "winapi-util", ] [[package]] name = "indexmap" version = "1.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1885e79c1fc4b10f0e172c475f458b7f7b93061064d98c3293e98c5ba0c8b399" dependencies = [ "autocfg", "hashbrown", ] [[package]] name = "instant" version = "0.1.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7a5bbe824c507c5da5956355e86a746d82e0e1464f65d862cc5e71da70e94b2c" dependencies = [ "cfg-if", ] [[package]] name = "itertools" version = "0.10.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b0fd2260e829bddf4cb6ea802289de2f86d6a7a690192fbe91b3f46e0f2c8473" dependencies = [ "either", ] [[package]] name = "itoa" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4217ad341ebadf8d8e724e264f13e593e0648f5b3e94b3896a5df283be015ecc" [[package]] name = "js-sys" version = "0.3.60" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "49409df3e3bf0856b916e2ceaca09ee28e6871cf7d9ce97a692cacfdb2a25a47" dependencies = [ "wasm-bindgen", ] [[package]] name = "jwalk" version = "0.8.1" dependencies = [ "criterion", "crossbeam", "fs_extra", "ignore", "lazy_static", "num_cpus", "rayon", "tempfile", "walkdir", ] [[package]] name = "lazy_static" version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e2abad23fbc42b3700f2f279844dc832adb2b2eb069b2df918f455c4e18cc646" [[package]] name = "libc" version = "0.2.138" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db6d7e329c562c5dfab7a46a2afabc8b987ab9a4834c9d1ca04dc54c1546cef8" [[package]] name = "log" version = "0.4.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "abb12e687cfb44aa40f41fc3978ef76448f9b6038cad6aef4259d3c095a2382e" dependencies = [ "cfg-if", ] [[package]] name = "memchr" version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2dffe52ecf27772e601905b7522cb4ef790d2cc203488bbd0e2fe85fcb74566d" [[package]] name = "memoffset" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5de893c32cde5f383baa4c04c5d6dbdd735cfd4a794b0debdb2bb1b421da5ff4" dependencies = [ "autocfg", ] [[package]] name = "num-traits" version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "578ede34cf02f8924ab9447f50c28075b4d3e5b269972345e7e0372b38c6cdcd" dependencies = [ "autocfg", ] [[package]] name = "num_cpus" version = "1.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f6058e64324c71e02bc2b150e4f3bc8286db6c83092132ffa3f6b1eab0f9def5" dependencies = [ "hermit-abi", "libc", ] [[package]] name = "once_cell" version = "1.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "86f0b0d4bf799edbc74508c1e8bf170ff5f41238e5f8225603ca7caaae2b7860" [[package]] name = "oorandom" version = "11.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ab1bc2a289d34bd04a330323ac98a1b4bc82c9d9fcb1e66b63caa84da26b575" [[package]] name = "os_str_bytes" version = "6.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b7820b9daea5457c9f21c69448905d723fbd21136ccf521748f23fd49e723ee" [[package]] name = "plotters" version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2538b639e642295546c50fcd545198c9d64ee2a38620a628724a3b266d5fbf97" dependencies = [ "num-traits", "plotters-backend", "plotters-svg", "wasm-bindgen", "web-sys", ] [[package]] name = "plotters-backend" version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "193228616381fecdc1224c62e96946dfbc73ff4384fba576e052ff8c1bea8142" [[package]] name = "plotters-svg" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f9a81d2759aae1dae668f783c308bc5c8ebd191ff4184aaa1b37f65a6ae5a56f" dependencies = [ "plotters-backend", ] [[package]] name = "proc-macro2" version = "1.0.47" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5ea3d908b0e36316caf9e9e2c4625cdde190a7e6f440d794667ed17a1855e725" dependencies = [ "unicode-ident", ] [[package]] name = "quote" version = "1.0.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bbe448f377a7d6961e30f5955f9b8d106c3f5e449d493ee1b125c1d43c2b5179" dependencies = [ "proc-macro2", ] [[package]] name = "rayon" version = "1.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6db3a213adf02b3bcfd2d3846bb41cb22857d131789e01df434fb7e7bc0759b7" dependencies = [ "either", "rayon-core", ] [[package]] name = "rayon-core" version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cac410af5d00ab6884528b4ab69d1e8e146e8d471201800fa1b4524126de6ad3" dependencies = [ "crossbeam-channel", "crossbeam-deque", "crossbeam-utils", "num_cpus", ] [[package]] name = "redox_syscall" version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fb5a58c1855b4b6819d59012155603f0b22ad30cad752600aadfcb695265519a" dependencies = [ "bitflags", ] [[package]] name = "regex" version = "1.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e076559ef8e241f2ae3479e36f97bd5741c0330689e217ad51ce2c76808b868a" dependencies = [ "aho-corasick", "memchr", "regex-syntax", ] [[package]] name = "regex-syntax" version = "0.6.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "456c603be3e8d448b072f410900c09faf164fbce2d480456f50eea6e25f9c848" [[package]] name = "remove_dir_all" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3acd125665422973a33ac9d3dd2df85edad0f4ae9b00dafb1a05e43a9f5ef8e7" dependencies = [ "winapi", ] [[package]] name = "ryu" version = "1.0.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4501abdff3ae82a1c1b477a17252eb69cee9e66eb915c1abaa4f44d873df9f09" [[package]] name = "same-file" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" dependencies = [ "winapi-util", ] [[package]] name = "scopeguard" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d29ab0c6d3fc0ee92fe66e2d99f700eab17a8d57d1c1d3b748380fb20baa78cd" [[package]] name = "serde" version = "1.0.150" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e326c9ec8042f1b5da33252c8a37e9ffbd2c9bef0155215b6e6c80c790e05f91" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" version = "1.0.150" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "42a3df25b0713732468deadad63ab9da1f1fd75a48a15024b50363f128db627e" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "serde_json" version = "1.0.89" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "020ff22c755c2ed3f8cf162dbb41a7268d934702f3ed3631656ea597e08fc3db" dependencies = [ "itoa", "ryu", "serde", ] [[package]] name = "syn" version = "1.0.105" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "60b9b43d45702de4c839cb9b51d9f529c5dd26a4aff255b42b1ebc03e88ee908" dependencies = [ "proc-macro2", "quote", "unicode-ident", ] [[package]] name = "tempfile" version = "3.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5cdb1ef4eaeeaddc8fbd371e5017057064af0911902ef36b39801f67cc6d79e4" dependencies = [ "cfg-if", "fastrand", "libc", "redox_syscall", "remove_dir_all", "winapi", ] [[package]] name = "textwrap" version = "0.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "222a222a5bfe1bba4a77b45ec488a741b3cb8872e5e499451fd7d0129c9c7c3d" [[package]] name = "thread_local" version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5516c27b78311c50bf42c071425c560ac799b11c30b31f87e3081965fe5e0180" dependencies = [ "once_cell", ] [[package]] name = "tinytemplate" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc" dependencies = [ "serde", "serde_json", ] [[package]] name = "unicode-ident" version = "1.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ceab39d59e4c9499d4e5a8ee0e2735b891bb7308ac83dfb4e80cad195c9f6f3" [[package]] name = "walkdir" version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "808cf2735cd4b6866113f648b791c6adc5714537bc222d9347bb203386ffda56" dependencies = [ "same-file", "winapi", "winapi-util", ] [[package]] name = "wasm-bindgen" version = "0.2.83" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eaf9f5aceeec8be17c128b2e93e031fb8a4d469bb9c4ae2d7dc1888b26887268" dependencies = [ "cfg-if", "wasm-bindgen-macro", ] [[package]] name = "wasm-bindgen-backend" version = "0.2.83" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c8ffb332579b0557b52d268b91feab8df3615f265d5270fec2a8c95b17c1142" dependencies = [ "bumpalo", "log", "once_cell", "proc-macro2", "quote", "syn", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-macro" version = "0.2.83" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "052be0f94026e6cbc75cdefc9bae13fd6052cdcaf532fa6c45e7ae33a1e6c810" dependencies = [ "quote", "wasm-bindgen-macro-support", ] [[package]] name = "wasm-bindgen-macro-support" version = "0.2.83" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "07bc0c051dc5f23e307b13285f9d75df86bfdf816c5721e573dec1f9b8aa193c" dependencies = [ "proc-macro2", "quote", "syn", "wasm-bindgen-backend", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" version = "0.2.83" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1c38c045535d93ec4f0b4defec448e4291638ee608530863b1e2ba115d4fff7f" [[package]] name = "web-sys" version = "0.3.60" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bcda906d8be16e728fd5adc5b729afad4e444e106ab28cd1c7256e54fa61510f" dependencies = [ "js-sys", "wasm-bindgen", ] [[package]] name = "winapi" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" dependencies = [ "winapi-i686-pc-windows-gnu", "winapi-x86_64-pc-windows-gnu", ] [[package]] name = "winapi-i686-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" [[package]] name = "winapi-util" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "70ec6ce85bb158151cae5e5c87f95a8e97d2c0c4b001223f33a334e3ce5de178" dependencies = [ "winapi", ] [[package]] name = "winapi-x86_64-pc-windows-gnu" version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" jwalk-0.8.1/Cargo.toml0000644000000027170000000000100101570ustar # THIS FILE IS AUTOMATICALLY GENERATED BY CARGO # # When uploading crates to the registry Cargo will automatically # "normalize" Cargo.toml files for maximal compatibility # with all versions of Cargo and also rewrite `path` dependencies # to registry (e.g., crates.io) dependencies. # # If you are reading this file be aware that the original Cargo.toml # will likely look very different (and much more reasonable). # See Cargo.toml.orig for the original contents. [package] edition = "2018" name = "jwalk" version = "0.8.1" authors = [ "Jesse Grosjean ", "Sebastian Thiel ", ] description = "Filesystem walk performed in parallel with streamed and sorted results." homepage = "https://github.com/byron/jwalk" documentation = "https://docs.rs/jwalk/" readme = "README.md" keywords = [ "directory", "recursive", "walk", "iterator", "parallel", ] categories = [ "filesystem", "concurrency", ] license = "MIT" repository = "https://github.com/byron/jwalk" [[bench]] name = "walk_benchmark" harness = false [dependencies.crossbeam] version = "0.8" [dependencies.rayon] version = "1.5" [dev-dependencies.criterion] version = "0.4.0" [dev-dependencies.fs_extra] version = "1.2" [dev-dependencies.ignore] version = "0.4" [dev-dependencies.lazy_static] version = "1.4" [dev-dependencies.num_cpus] version = "1.12" [dev-dependencies.tempfile] version = "3.1" [dev-dependencies.walkdir] version = "2.3" jwalk-0.8.1/Cargo.toml.orig0000644000000013770000000000100111170ustar [package] name = "jwalk" version = "0.8.1" authors = ["Jesse Grosjean ", "Sebastian Thiel "] description = "Filesystem walk performed in parallel with streamed and sorted results." documentation = "https://docs.rs/jwalk/" homepage = "https://github.com/byron/jwalk" repository = "https://github.com/byron/jwalk" readme = "README.md" keywords = ["directory", "recursive", "walk", "iterator", "parallel"] categories = ["filesystem", "concurrency"] license = "MIT" edition = "2018" [dependencies] rayon = "1.5" crossbeam = "0.8" [dev-dependencies] criterion = "0.4.0" fs_extra = "1.2" walkdir = "2.3" ignore = "0.4" tempfile = "3.1" num_cpus = "1.12" lazy_static = "1.4" [[bench]] name = "walk_benchmark" harness = false jwalk-0.8.1/Cargo.toml.orig000064400000000000000000000013771046102023000136410ustar 00000000000000[package] name = "jwalk" version = "0.8.1" authors = ["Jesse Grosjean ", "Sebastian Thiel "] description = "Filesystem walk performed in parallel with streamed and sorted results." documentation = "https://docs.rs/jwalk/" homepage = "https://github.com/byron/jwalk" repository = "https://github.com/byron/jwalk" readme = "README.md" keywords = ["directory", "recursive", "walk", "iterator", "parallel"] categories = ["filesystem", "concurrency"] license = "MIT" edition = "2018" [dependencies] rayon = "1.5" crossbeam = "0.8" [dev-dependencies] criterion = "0.4.0" fs_extra = "1.2" walkdir = "2.3" ignore = "0.4" tempfile = "3.1" num_cpus = "1.12" lazy_static = "1.4" [[bench]] name = "walk_benchmark" harness = false jwalk-0.8.1/LICENSE000064400000000000000000000020701046102023000117460ustar 00000000000000The MIT License (MIT) Copyright (c) 2019 Jesse Grosjean Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.jwalk-0.8.1/README.md000064400000000000000000000034061046102023000122240ustar 00000000000000jwalk ======= Filesystem walk. - Performed in parallel using rayon - Entries streamed in sorted order - Custom sort/filter/skip/state [![Build Status](https://travis-ci.org/Byron/jwalk.svg?branch=main)](https://travis-ci.org/Byron/jwalk) [![Latest version](http://meritbadge.herokuapp.com/jwalk)](https://crates.io/crates/jwalk) ### Usage Add this to your `Cargo.toml`: ```toml [dependencies] jwalk = "0.5" ``` Lean More: [docs.rs/jwalk](https://docs.rs/jwalk) ### Example Recursively iterate over the "foo" directory sorting by name: ```rust use jwalk::{WalkDir}; for entry in WalkDir::new("foo").sort(true) { println!("{}", entry?.path().display()); } ``` ### Inspiration This crate is inspired by both [`walkdir`](https://crates.io/crates/walkdir) and [`ignore`](https://crates.io/crates/ignore). It attempts to combine the parallelism of `ignore` with `walkdir`'s streaming iterator API. Some code and comments are copied directly from `walkdir`. ### Why use this crate? This crate is particularly good when you want streamed sorted results. In my tests it's about 4x `walkdir` speed for sorted results with metadata. Also this crate's `process_read_dir` callback allows you to arbitrarily sort/filter/skip/state entries before they are yielded. ### Why not use this crate? Directory traversal is already pretty fast. If you don't need this crate's speed then `walkdir` provides a smaller and more tested single threaded implementation. This crates parallelism happens at the directory level. It will help when walking deep file systems with many directories. It wont help when reading a single directory with many files. ### Benchmarks [Benchmarks](https://github.com/jessegrosjean/jwalk/blob/main/benches/benchmarks.md) comparing this crate with `walkdir` and `ignore`.jwalk-0.8.1/examples/du.rs000064400000000000000000000023211046102023000135340ustar 00000000000000extern crate jwalk; use jwalk::{Parallelism, WalkDirGeneric}; use std::env; fn main() { let path = env::args().skip(1).next().unwrap_or("./".to_owned()); let mut total: u64 = 0; for dir_entry_result in WalkDirGeneric::<((), Option)>::new(&path) .skip_hidden(false) .parallelism(Parallelism::RayonNewPool(4)) .process_read_dir(|_, _, _, dir_entry_results| { dir_entry_results.iter_mut().for_each(|dir_entry_result| { if let Ok(dir_entry) = dir_entry_result { if !dir_entry.file_type.is_dir() { dir_entry.client_state = Some(dir_entry.metadata().map(|m| m.len()).unwrap_or_default()); } } }) }) { match dir_entry_result { Ok(dir_entry) => { if let Some(len) = &dir_entry.client_state { eprintln!("counting {:?}", dir_entry.path()); total += len; } } Err(error) => { println!("Read dir_entry error: {}", error); } } } println!("path: {} total bytes: {}", path, total); } jwalk-0.8.1/src/core/dir_entry.rs000064400000000000000000000205301046102023000150260ustar 00000000000000use std::ffi::{OsStr, OsString}; use std::fmt; use std::fs::{self, FileType}; use std::path::{Path, PathBuf}; use std::sync::Arc; use crate::{ClientState, Error, ReadDirSpec, Result}; /// Representation of a file or directory. /// /// This representation does not wrap a `std::fs::DirEntry`. Instead it copies /// `file_name`, `file_type`, and optionally `metadata` out of the underlying /// `std::fs::DirEntry`. This allows it to quickly drop the underlying file /// descriptor. pub struct DirEntry { /// Depth of this entry relative to the root directory where the walk /// started. pub depth: usize, /// File name of this entry without leading path component. pub file_name: OsString, /// File type for the file/directory that this entry points at. pub file_type: FileType, /// Field where clients can store state from within the The /// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) /// callback. pub client_state: C::DirEntryState, /// Path used by this entry's parent to read this entry. pub parent_path: Arc, /// Path that will be used to read child entries. This is automatically set /// for directories. The /// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) callback /// may set this field to `None` to skip reading the contents of a /// particular directory. pub read_children_path: Option>, /// If `read_children_path` is set and resulting `fs::read_dir` generates an error /// then that error is stored here. pub read_children_error: Option, // True if [`follow_links`] is `true` AND was created from a symlink path. follow_link: bool, // Origins of symlinks followed to get to this entry. follow_link_ancestors: Arc>>, } impl DirEntry { pub(crate) fn from_entry( depth: usize, parent_path: Arc, fs_dir_entry: &fs::DirEntry, follow_link_ancestors: Arc>>, ) -> Result { let file_type = fs_dir_entry .file_type() .map_err(|err| Error::from_path(depth, fs_dir_entry.path(), err))?; let file_name = fs_dir_entry.file_name(); let read_children_path: Option> = if file_type.is_dir() { Some(Arc::from(parent_path.join(&file_name))) } else { None }; Ok(DirEntry { depth, file_name, file_type, parent_path, read_children_path, read_children_error: None, client_state: C::DirEntryState::default(), follow_link: false, follow_link_ancestors, }) } // Only used for root and when following links. pub(crate) fn from_path( depth: usize, path: &Path, follow_link: bool, follow_link_ancestors: Arc>>, ) -> Result { let metadata = if follow_link { fs::metadata(path).map_err(|err| Error::from_path(depth, path.to_owned(), err))? } else { fs::symlink_metadata(path) .map_err(|err| Error::from_path(depth, path.to_owned(), err))? }; let root_name = path.file_name().unwrap_or(path.as_os_str()); let read_children_path: Option> = if metadata.file_type().is_dir() { Some(Arc::from(path)) } else { None }; Ok(DirEntry { depth, file_name: root_name.to_owned(), file_type: metadata.file_type(), parent_path: Arc::from(path.parent().map(Path::to_path_buf).unwrap_or_default()), read_children_path, read_children_error: None, client_state: C::DirEntryState::default(), follow_link, follow_link_ancestors, }) } /// Return the file type for the file that this entry points to. /// /// If this is a symbolic link and [`follow_links`] is `true`, then this /// returns the type of the target. /// /// This never makes any system calls. /// /// [`follow_links`]: struct.WalkDir.html#method.follow_links pub fn file_type(&self) -> FileType { self.file_type } /// Return the file name of this entry. /// /// If this entry has no file name (e.g., `/`), then the full path is /// returned. pub fn file_name(&self) -> &OsStr { &self.file_name } /// Returns the depth at which this entry was created relative to the root. /// /// The smallest depth is `0` and always corresponds to the path given /// to the `new` function on `WalkDir`. Its direct descendants have depth /// `1`, and their descendants have depth `2`, and so on. pub fn depth(&self) -> usize { self.depth } /// Path to the file/directory represented by this entry. /// /// The path is created by joining `parent_path` with `file_name`. pub fn path(&self) -> PathBuf { self.parent_path.join(&self.file_name) } /// Returns `true` if and only if this entry was created from a symbolic /// link. This is unaffected by the [`follow_links`] setting. /// /// When `true`, the value returned by the [`path`] method is a /// symbolic link name. To get the full target path, you must call /// [`std::fs::read_link(entry.path())`]. /// /// [`path`]: struct.DirEntry.html#method.path /// [`follow_links`]: struct.WalkDir.html#method.follow_links /// [`std::fs::read_link(entry.path())`]: https://doc.rust-lang.org/stable/std/fs/fn.read_link.html pub fn path_is_symlink(&self) -> bool { self.file_type.is_symlink() || self.follow_link } /// Return the metadata for the file that this entry points to. /// /// This will follow symbolic links if and only if the [`WalkDir`] value /// has [`follow_links`] enabled. /// /// # Platform behavior /// /// This always calls [`std::fs::symlink_metadata`]. /// /// If this entry is a symbolic link and [`follow_links`] is enabled, then /// [`std::fs::metadata`] is called instead. /// /// # Errors /// /// Similar to [`std::fs::metadata`], returns errors for path values that /// the program does not have permissions to access or if the path does not /// exist. /// /// [`WalkDir`]: struct.WalkDir.html /// [`follow_links`]: struct.WalkDir.html#method.follow_links /// [`std::fs::metadata`]: https://doc.rust-lang.org/std/fs/fn.metadata.html /// [`std::fs::symlink_metadata`]: https://doc.rust-lang.org/stable/std/fs/fn.symlink_metadata.html pub fn metadata(&self) -> Result { if self.follow_link { fs::metadata(&self.path()) } else { fs::symlink_metadata(&self.path()) } .map_err(|err| Error::from_entry(self, err)) } /// Reference to the path of the directory containing this entry. pub fn parent_path(&self) -> &Path { &self.parent_path } pub(crate) fn read_children_spec( &self, client_read_state: C::ReadDirState, ) -> Option> { self.read_children_path .as_ref() .map(|read_children_path| ReadDirSpec { depth: self.depth, client_read_state, path: read_children_path.clone(), follow_link_ancestors: self.follow_link_ancestors.clone(), }) } pub(crate) fn follow_symlink(&self) -> Result { let path = self.path(); let origins = self.follow_link_ancestors.clone(); let dir_entry = DirEntry::from_path(self.depth, &path, true, origins)?; if dir_entry.file_type.is_dir() { let target = fs::read_link(&path).map_err(|err| Error::from_io(self.depth, err))?; for ancestor in self.follow_link_ancestors.iter().rev() { if target.as_path() == ancestor.as_ref() { return Err(Error::from_loop( self.depth, ancestor.as_ref(), path.as_ref(), )); } } } Ok(dir_entry) } } impl fmt::Debug for DirEntry { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { write!(f, "DirEntry({:?})", self.path()) } } jwalk-0.8.1/src/core/dir_entry_iter.rs000064400000000000000000000076001046102023000160540ustar 00000000000000use std::iter::Peekable; use super::*; use crate::Result; /// DirEntry iterator from `WalkDir.into_iter()`. /// /// Yields entries from recursive traversal of filesystem. pub struct DirEntryIter { min_depth: usize, // iterator yielding next ReadDir results when needed pub(crate) read_dir_iter: Option>>, // stack of ReadDir results, track location in filesystem traversal read_dir_results_stack: Vec>>>, } impl DirEntryIter { pub(crate) fn new( root_entry_results: Vec>>, parallelism: Parallelism, min_depth: usize, root_read_dir_state: C::ReadDirState, core_read_dir_callback: Arc>, ) -> DirEntryIter { // 1. Gather read_dir_specs from root level let read_dir_specs: Vec<_> = root_entry_results .iter() .flat_map(|dir_entry_result| { dir_entry_result .as_ref() .ok()? .read_children_spec(root_read_dir_state.clone()) }) .collect(); // 2. Init new read_dir_iter from those specs let read_dir_iter = ReadDirIter::try_new(read_dir_specs, parallelism, core_read_dir_callback) .map(|iter| iter.peekable()); // 3. Return DirEntryIter that will return initial root entries and then // fill and process read_dir_iter until complete DirEntryIter { min_depth, read_dir_iter, read_dir_results_stack: vec![root_entry_results.into_iter()], } } fn push_next_read_dir_results( iter: &mut Peekable>, results: &mut Vec>>>, ) -> Result<()> { // Push next read dir results or return error if read failed let read_dir_result = iter.next().unwrap(); let read_dir = match read_dir_result { Ok(read_dir) => read_dir, Err(err) => return Err(err), }; let ReadDir { results_list, .. } = read_dir; results.push(results_list.into_iter()); Ok(()) } } impl Iterator for DirEntryIter { type Item = Result>; fn next(&mut self) -> Option { loop { // 1. Get current read dir results iter from top of stack let top_read_dir_results = self.read_dir_results_stack.last_mut()?; // 2. If more results in current read dir then process if let Some(dir_entry_result) = top_read_dir_results.next() { // 2.1 Handle error case let mut dir_entry = match dir_entry_result { Ok(dir_entry) => dir_entry, Err(err) => return Some(Err(err)), }; // 2.2 If dir_entry has a read_children_path means we need to read a new // directory and push those results onto read_dir_results_stack if dir_entry.read_children_path.is_some() { let iter = match self.read_dir_iter.as_mut().ok_or_else(Error::busy) { Ok(iter) => iter, Err(err) => return Some(Err(err)), }; if let Err(err) = Self::push_next_read_dir_results(iter, &mut self.read_dir_results_stack) { dir_entry.read_children_error = Some(err); } } if dir_entry.depth >= self.min_depth { // 2.3 Finished, return dir_entry return Some(Ok(dir_entry)); } } else { // If no more results in current then pop stack self.read_dir_results_stack.pop(); } } } } jwalk-0.8.1/src/core/error.rs000064400000000000000000000242271046102023000141670ustar 00000000000000use std::error; use std::fmt; use std::io; use std::path::{Path, PathBuf}; use crate::{ClientState, DirEntry}; /// An error produced by recursively walking a directory. /// /// This error type is a light wrapper around [`std::io::Error`]. In /// particular, it adds the following information: /// /// * The depth at which the error occurred in the file tree, relative to the /// root. /// * The path, if any, associated with the IO error. /// * An indication that a loop occurred when following symbolic links. In this /// case, there is no underlying IO error. /// /// To maintain good ergonomics, this type has a /// [`impl From for std::io::Error`][impl] defined which preserves the original context. /// This allows you to use an [`io::Result`] with methods in this crate if you don't care about /// accessing the underlying error data in a structured form. /// /// [`std::io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html /// [`io::Result`]: https://doc.rust-lang.org/stable/std/io/type.Result.html /// [impl]: struct.Error.html#impl-From%3CError%3E #[derive(Debug)] pub struct Error { depth: usize, inner: ErrorInner, } #[derive(Debug)] enum ErrorInner { Io { path: Option, err: io::Error, }, Loop { ancestor: PathBuf, child: PathBuf, }, ThreadpoolBusy, } impl Error { /// Returns the path associated with this error if one exists. /// /// For example, if an error occurred while opening a directory handle, /// the error will include the path passed to [`std::fs::read_dir`]. /// /// [`std::fs::read_dir`]: https://doc.rust-lang.org/stable/std/fs/fn.read_dir.html pub fn path(&self) -> Option<&Path> { match self.inner { ErrorInner::ThreadpoolBusy => None, ErrorInner::Io { path: None, .. } => None, ErrorInner::Io { path: Some(ref path), .. } => Some(path), ErrorInner::Loop { ref child, .. } => Some(child), } } /// Returns the path at which a cycle was detected. /// /// If no cycle was detected, [`None`] is returned. /// /// A cycle is detected when a directory entry is equivalent to one of /// its ancestors. /// /// To get the path to the child directory entry in the cycle, use the /// [`path`] method. /// /// [`None`]: https://doc.rust-lang.org/stable/std/option/enum.Option.html#variant.None /// [`path`]: struct.Error.html#path pub fn loop_ancestor(&self) -> Option<&Path> { match self.inner { ErrorInner::Loop { ref ancestor, .. } => Some(ancestor), _ => None, } } /// Returns the depth at which this error occurred relative to the root. /// /// The smallest depth is `0` and always corresponds to the path given to /// the [`new`] function on [`WalkDir`]. Its direct descendants have depth /// `1`, and their descendants have depth `2`, and so on. /// /// [`new`]: struct.WalkDir.html#method.new /// [`WalkDir`]: struct.WalkDir.html pub fn depth(&self) -> usize { self.depth } /// Inspect the original [`io::Error`] if there is one. /// /// [`None`] is returned if the [`Error`] doesn't correspond to an /// [`io::Error`]. This might happen, for example, when the error was /// produced because a cycle was found in the directory tree while /// following symbolic links. /// /// This method returns a borrowed value that is bound to the lifetime of the [`Error`]. To /// obtain an owned value, the [`into_io_error`] can be used instead. /// /// > This is the original [`io::Error`] and is _not_ the same as /// > [`impl From for std::io::Error`][impl] which contains additional context about the /// error. /// /// # Example /// /// ```rust,no_run /// use std::io; /// use std::path::Path; /// /// use walkdir::WalkDir; /// /// for entry in WalkDir::new("foo") { /// match entry { /// Ok(entry) => println!("{}", entry.path().display()), /// Err(err) => { /// let path = err.path().unwrap_or(Path::new("")).display(); /// println!("failed to access entry {}", path); /// if let Some(inner) = err.io_error() { /// match inner.kind() { /// io::ErrorKind::InvalidData => { /// println!( /// "entry contains invalid data: {}", /// inner) /// } /// io::ErrorKind::PermissionDenied => { /// println!( /// "Missing permission to read entry: {}", /// inner) /// } /// _ => { /// println!( /// "Unexpected error occurred: {}", /// inner) /// } /// } /// } /// } /// } /// } /// ``` /// /// [`None`]: https://doc.rust-lang.org/stable/std/option/enum.Option.html#variant.None /// [`io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html /// [`From`]: https://doc.rust-lang.org/stable/std/convert/trait.From.html /// [`Error`]: struct.Error.html /// [`into_io_error`]: struct.Error.html#method.into_io_error /// [impl]: struct.Error.html#impl-From%3CError%3E pub fn io_error(&self) -> Option<&io::Error> { match self.inner { ErrorInner::Io { ref err, .. } => Some(err), _ => None, } } /// Returns true if this error is due to a busy thread-pool that prevented its effective use. /// /// Note that business detection is timeout based, and we don't know if it would have been a deadlock or not. pub fn is_busy(&self) -> bool { matches!(self.inner, ErrorInner::ThreadpoolBusy) } /// Similar to [`io_error`] except consumes self to convert to the original /// [`io::Error`] if one exists. /// /// [`io_error`]: struct.Error.html#method.io_error /// [`io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html pub fn into_io_error(self) -> Option { match self.inner { ErrorInner::Io { err, .. } => Some(err), _ => None, } } pub(crate) fn busy() -> Self { Error { depth: 0, inner: ErrorInner::ThreadpoolBusy, } } pub(crate) fn from_path(depth: usize, pb: PathBuf, err: io::Error) -> Self { Error { depth, inner: ErrorInner::Io { path: Some(pb), err, }, } } pub(crate) fn from_entry(dent: &DirEntry, err: io::Error) -> Self { Error { depth: dent.depth(), inner: ErrorInner::Io { path: Some(dent.path()), err, }, } } pub(crate) fn from_io(depth: usize, err: io::Error) -> Self { Error { depth, inner: ErrorInner::Io { path: None, err }, } } pub(crate) fn from_loop(depth: usize, ancestor: &Path, child: &Path) -> Self { Error { depth, inner: ErrorInner::Loop { ancestor: ancestor.to_path_buf(), child: child.to_path_buf(), }, } } } impl error::Error for Error { fn source(&self) -> Option<&(dyn error::Error + 'static)> { match self.inner { ErrorInner::Io { ref err, .. } => Some(err), ErrorInner::Loop { .. } | ErrorInner::ThreadpoolBusy => None, } } #[allow(deprecated)] fn description(&self) -> &str { match self.inner { ErrorInner::Io { ref err, .. } => err.description(), ErrorInner::Loop { .. } => "file system loop found", ErrorInner::ThreadpoolBusy => "thread-pool busy", } } fn cause(&self) -> Option<&dyn error::Error> { self.source() } } impl fmt::Display for Error { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { match self.inner { ErrorInner::ThreadpoolBusy => f.write_str("rayon thread-pool too busy or dependency loop detected - aborting before possibility of deadlock"), ErrorInner::Io { path: None, ref err, } => err.fmt(f), ErrorInner::Io { path: Some(ref path), ref err, } => write!(f, "IO error for operation on {}: {}", path.display(), err), ErrorInner::Loop { ref ancestor, ref child, } => write!( f, "File system loop found: \ {} points to an ancestor {}", child.display(), ancestor.display() ), } } } impl From for io::Error { /// Convert the [`Error`] to an [`io::Error`], preserving the original /// [`Error`] as the ["inner error"]. Note that this also makes the display /// of the error include the context. /// /// This is different from [`into_io_error`] which returns the original /// [`io::Error`]. /// /// [`Error`]: struct.Error.html /// [`io::Error`]: https://doc.rust-lang.org/stable/std/io/struct.Error.html /// ["inner error"]: https://doc.rust-lang.org/std/io/struct.Error.html#method.into_inner /// [`into_io_error`]: struct.WalkDir.html#method.into_io_error fn from(walk_err: Error) -> io::Error { let kind = match walk_err { Error { inner: ErrorInner::Io { ref err, .. }, .. } => err.kind(), Error { inner: ErrorInner::Loop { .. }, .. } => io::ErrorKind::Other, Error { inner: ErrorInner::ThreadpoolBusy, .. } => io::ErrorKind::Other, }; io::Error::new(kind, walk_err) } } jwalk-0.8.1/src/core/index_path.rs000064400000000000000000000021251046102023000151520ustar 00000000000000use std::cmp::Ordering; #[derive(Clone, Debug)] pub struct IndexPath { pub indices: Vec, } impl IndexPath { pub fn new(indices: Vec) -> IndexPath { IndexPath { indices } } pub fn adding(&self, index: usize) -> IndexPath { let mut indices = self.indices.clone(); indices.push(index); IndexPath::new(indices) } pub fn push(&mut self, index: usize) { self.indices.push(index); } pub fn increment_last(&mut self) { *self.indices.last_mut().unwrap() += 1; } pub fn pop(&mut self) -> Option { self.indices.pop() } pub fn is_empty(&self) -> bool { self.indices.is_empty() } } impl PartialEq for IndexPath { fn eq(&self, o: &Self) -> bool { self.indices.eq(&o.indices) } } impl Eq for IndexPath {} impl PartialOrd for IndexPath { fn partial_cmp(&self, o: &Self) -> Option { o.indices.partial_cmp(&self.indices) } } impl Ord for IndexPath { fn cmp(&self, o: &Self) -> Ordering { o.indices.cmp(&self.indices) } } jwalk-0.8.1/src/core/mod.rs000064400000000000000000000010451046102023000136060ustar 00000000000000mod dir_entry; mod dir_entry_iter; mod error; mod index_path; mod ordered; mod ordered_queue; mod read_dir; mod read_dir_iter; mod read_dir_spec; mod run_context; use rayon::prelude::*; use std::sync::atomic::AtomicBool; use std::sync::Arc; use std::vec; use index_path::*; use ordered::*; use ordered_queue::*; use read_dir_iter::*; use run_context::*; pub use dir_entry::DirEntry; pub use dir_entry_iter::DirEntryIter; pub use error::Error; pub use read_dir::ReadDir; pub use read_dir_spec::ReadDirSpec; use crate::{ClientState, Parallelism}; jwalk-0.8.1/src/core/ordered.rs000064400000000000000000000014521046102023000144550ustar 00000000000000use std::cmp::Ordering; use super::index_path::IndexPath; pub struct Ordered { pub value: T, pub index_path: IndexPath, pub(crate) child_count: usize, } impl Ordered { pub fn new(value: T, index_path: IndexPath, child_count: usize) -> Ordered { Ordered { value, index_path, child_count, } } } impl PartialEq for Ordered { fn eq(&self, o: &Self) -> bool { self.index_path.eq(&o.index_path) } } impl Eq for Ordered {} impl PartialOrd for Ordered { fn partial_cmp(&self, o: &Self) -> Option { self.index_path.partial_cmp(&o.index_path) } } impl Ord for Ordered { fn cmp(&self, o: &Self) -> Ordering { self.index_path.cmp(&o.index_path) } } jwalk-0.8.1/src/core/ordered_queue.rs000064400000000000000000000131061046102023000156600ustar 00000000000000//! Ordered queue backed by a channel. use crossbeam::channel::{self, Receiver, SendError, Sender, TryRecvError}; use std::collections::BinaryHeap; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering as AtomicOrdering}; use std::sync::Arc; use std::thread; use super::*; pub(crate) struct OrderedQueue where T: Send, { sender: Sender>, pending_count: Arc, stop: Arc, } pub enum Ordering { Relaxed, Strict, } pub struct OrderedQueueIter where T: Send, { ordering: Ordering, stop: Arc, receiver: Receiver>, receive_buffer: BinaryHeap>, pending_count: Arc, ordered_matcher: OrderedMatcher, } struct OrderedMatcher { looking_for: IndexPath, child_count_stack: Vec, } pub(crate) fn new_ordered_queue( stop: Arc, ordering: Ordering, ) -> (OrderedQueue, OrderedQueueIter) where T: Send, { let pending_count = Arc::new(AtomicUsize::new(0)); let (sender, receiver) = channel::unbounded(); ( OrderedQueue { sender, pending_count: pending_count.clone(), stop: stop.clone(), }, OrderedQueueIter { ordering, receiver, ordered_matcher: OrderedMatcher::default(), receive_buffer: BinaryHeap::new(), pending_count, stop, }, ) } impl OrderedQueue where T: Send, { pub fn push(&self, ordered: Ordered) -> Result<(), SendError>> { self.pending_count.fetch_add(1, AtomicOrdering::SeqCst); self.sender.send(ordered) } pub fn complete_item(&self) { self.pending_count.fetch_sub(1, AtomicOrdering::SeqCst); } } impl Clone for OrderedQueue where T: Send, { fn clone(&self) -> Self { OrderedQueue { sender: self.sender.clone(), pending_count: self.pending_count.clone(), stop: self.stop.clone(), } } } impl OrderedQueueIter where T: Send, { fn pending_count(&self) -> usize { self.pending_count.load(AtomicOrdering::SeqCst) } fn is_stop(&self) -> bool { self.stop.load(AtomicOrdering::SeqCst) } fn try_next_relaxed(&mut self) -> Result, TryRecvError> { if self.is_stop() { return Err(TryRecvError::Disconnected); } while let Ok(ordered_work) = self.receiver.try_recv() { self.receive_buffer.push(ordered_work) } if let Some(ordered_work) = self.receive_buffer.pop() { Ok(ordered_work) } else if self.pending_count() == 0 { Err(TryRecvError::Disconnected) } else { Err(TryRecvError::Empty) } } fn try_next_strict(&mut self) -> Result, TryRecvError> { let looking_for = &self.ordered_matcher.looking_for; loop { if self.is_stop() { return Err(TryRecvError::Disconnected); } let top_ordered = self.receive_buffer.peek(); if let Some(top_ordered) = top_ordered { if top_ordered.index_path.eq(looking_for) { break; } } if self.ordered_matcher.is_none() { return Err(TryRecvError::Disconnected); } match self.receiver.try_recv() { Ok(ordered) => { self.receive_buffer.push(ordered); } Err(err) => match err { TryRecvError::Empty => thread::yield_now(), TryRecvError::Disconnected => break, }, } } let ordered = self.receive_buffer.pop().unwrap(); self.ordered_matcher.advance_past(&ordered); Ok(ordered) } } impl Iterator for OrderedQueueIter where T: Send, { type Item = Ordered; fn next(&mut self) -> Option> { loop { let try_next = match self.ordering { Ordering::Relaxed => self.try_next_relaxed(), Ordering::Strict => self.try_next_strict(), }; match try_next { Ok(next) => { return Some(next); } Err(err) => match err { TryRecvError::Empty => thread::yield_now(), TryRecvError::Disconnected => return None, }, } } } } impl OrderedMatcher { fn is_none(&self) -> bool { self.looking_for.is_empty() } fn decrement_remaining_children(&mut self) { *self.child_count_stack.last_mut().unwrap() -= 1; } fn advance_past(&mut self, ordered: &Ordered) { self.decrement_remaining_children(); if ordered.child_count > 0 { self.looking_for.push(0); self.child_count_stack.push(ordered.child_count); } else { self.looking_for.increment_last(); while !self.child_count_stack.is_empty() && *self.child_count_stack.last().unwrap() == 0 { self.looking_for.pop(); self.child_count_stack.pop(); if !self.looking_for.is_empty() { self.looking_for.increment_last(); } } } } } impl Default for OrderedMatcher { fn default() -> OrderedMatcher { OrderedMatcher { looking_for: IndexPath::new(vec![0]), child_count_stack: vec![1], } } } jwalk-0.8.1/src/core/read_dir.rs000064400000000000000000000021331046102023000145770ustar 00000000000000use super::{ClientState, DirEntry, IndexPath, Ordered, ReadDirSpec}; use crate::Result; /// Results of successfully reading a directory. #[derive(Debug)] pub struct ReadDir { pub(crate) read_dir_state: C::ReadDirState, pub(crate) results_list: Vec>>, } impl ReadDir { pub fn new( read_dir_state: C::ReadDirState, results_list: Vec>>, ) -> ReadDir { ReadDir { read_dir_state, results_list, } } pub fn read_children_specs(&self) -> impl Iterator> + '_ { self.results_list.iter().filter_map(move |each| { each.as_ref() .ok()? .read_children_spec(self.read_dir_state.clone()) }) } pub fn ordered_read_children_specs( &self, index_path: &IndexPath, ) -> Vec>> { self.read_children_specs() .enumerate() .map(|(i, spec)| Ordered::new(spec, index_path.adding(i), 0)) .collect() } } jwalk-0.8.1/src/core/read_dir_iter.rs000064400000000000000000000122611046102023000156250ustar 00000000000000use std::sync::Arc; use super::*; use crate::Result; /// Client's read dir function. pub(crate) type ReadDirCallback = dyn Fn(ReadDirSpec) -> Result> + Send + Sync + 'static; /// Result Iterator. /// /// Yields ReadDirs (results of fs::read_dir) in order required for recursive /// directory traversal. Depending on Walk/ParWalk state these reads might be /// computed in parallel. pub enum ReadDirIter { Walk { read_dir_spec_stack: Vec>, core_read_dir_callback: Arc>, }, ParWalk { read_dir_result_iter: OrderedQueueIter>>, }, } impl ReadDirIter { pub(crate) fn try_new( read_dir_specs: Vec>, parallelism: Parallelism, core_read_dir_callback: Arc>, ) -> Option { if let Parallelism::Serial = parallelism { ReadDirIter::Walk { read_dir_spec_stack: read_dir_specs, core_read_dir_callback, } } else { let stop = Arc::new(AtomicBool::new(false)); let read_dir_result_queue = new_ordered_queue(stop.clone(), Ordering::Strict); let (read_dir_result_queue, read_dir_result_iter) = read_dir_result_queue; let read_dir_spec_queue = new_ordered_queue(stop.clone(), Ordering::Relaxed); let (read_dir_spec_queue, read_dir_spec_iter) = read_dir_spec_queue; for (i, read_dir_spec) in read_dir_specs.into_iter().enumerate() { read_dir_spec_queue .push(Ordered::new(read_dir_spec, IndexPath::new(vec![0]), i)) .unwrap(); } let run_context = RunContext { stop, read_dir_spec_queue, read_dir_result_queue, core_read_dir_callback, }; let (startup_tx, startup_rx) = parallelism .timeout() .map(|duration| { let (tx, rx) = crossbeam::channel::unbounded(); (Some(tx), Some((rx, duration))) }) .unwrap_or((None, None)); parallelism.spawn(move || { if let Some(tx) = startup_tx { if tx.send(()).is_err() { // rayon didn't install this function in time so the listener exited. Do the same. return; } } read_dir_spec_iter.par_bridge().for_each_with( run_context, |run_context, ordered_read_dir_spec| { multi_threaded_walk_dir(ordered_read_dir_spec, run_context); }, ); }); if startup_rx.map_or(false, |(rx, duration)| rx.recv_timeout(duration).is_err()) { return None; } ReadDirIter::ParWalk { read_dir_result_iter, } } .into() } } impl Iterator for ReadDirIter { type Item = Result>; fn next(&mut self) -> Option { match self { ReadDirIter::Walk { read_dir_spec_stack, core_read_dir_callback, } => { let read_dir_spec = read_dir_spec_stack.pop()?; let read_dir_result = core_read_dir_callback(read_dir_spec); if let Ok(read_dir) = read_dir_result.as_ref() { for each_spec in read_dir .read_children_specs() .collect::>() .into_iter() .rev() { read_dir_spec_stack.push(each_spec); } } Some(read_dir_result) } ReadDirIter::ParWalk { read_dir_result_iter, } => read_dir_result_iter .next() .map(|read_dir_result| read_dir_result.value), } } } fn multi_threaded_walk_dir( ordered_read_dir_spec: Ordered>, run_context: &mut RunContext, ) { let Ordered { value: read_dir_spec, index_path, .. } = ordered_read_dir_spec; let read_dir_result = (run_context.core_read_dir_callback)(read_dir_spec); let ordered_read_children_specs = read_dir_result .as_ref() .ok() .map(|read_dir| read_dir.ordered_read_children_specs(&index_path)); let ordered_read_dir_result = Ordered::new( read_dir_result, index_path, ordered_read_children_specs.as_ref().map_or(0, Vec::len), ); if !run_context.send_read_dir_result(ordered_read_dir_result) { run_context.stop(); return; } if let Some(ordered_read_children_specs) = ordered_read_children_specs { for each in ordered_read_children_specs { if !run_context.schedule_read_dir_spec(each) { run_context.stop(); return; } } } run_context.complete_item(); } jwalk-0.8.1/src/core/read_dir_spec.rs000064400000000000000000000021561046102023000156160ustar 00000000000000use std::path::Path; use std::sync::Arc; use crate::ClientState; /// Specification for reading a directory. /// /// When a directory is read a new `ReadDirSpec` is created for each folder /// found in that directory. These specs are then sent to a work queue that is /// used to schedule future directory reads. Use /// [`max_depth`](struct.WalkDir.html#method.max_depth) and /// [`process_read_dir`](struct.WalkDir.html#method.process_read_dir) to change /// this default behavior. #[derive(Debug)] pub struct ReadDirSpec { /// Depth of the directory to read relative to root of walk. pub depth: usize, /// Path of the the directory to read. pub path: Arc, /// Client branch state that was set in the /// [`process_read_dir`](struct.WalkDir.html#method.process_read_dir) callback /// when reading this directory's parent. One intended use case is to store /// `.gitignore` state to filter entries during the walk. pub client_read_state: C::ReadDirState, // Origins of symlinks followed to get to this entry. pub(crate) follow_link_ancestors: Arc>>, } jwalk-0.8.1/src/core/run_context.rs000064400000000000000000000026471046102023000154100ustar 00000000000000use std::sync::atomic::{AtomicBool, Ordering as AtomicOrdering}; use std::sync::Arc; use super::{ClientState, Ordered, OrderedQueue, ReadDir, ReadDirCallback, ReadDirSpec}; use crate::Result; pub(crate) struct RunContext { pub(crate) stop: Arc, pub(crate) read_dir_spec_queue: OrderedQueue>, pub(crate) read_dir_result_queue: OrderedQueue>>, pub(crate) core_read_dir_callback: Arc>, } impl RunContext { pub(crate) fn stop(&self) { self.stop.store(true, AtomicOrdering::SeqCst); } pub(crate) fn schedule_read_dir_spec(&self, ordered_read_dir: Ordered>) -> bool { self.read_dir_spec_queue.push(ordered_read_dir).is_ok() } pub(crate) fn send_read_dir_result( &self, read_dir_result: Ordered>>, ) -> bool { self.read_dir_result_queue.push(read_dir_result).is_ok() } pub(crate) fn complete_item(&self) { self.read_dir_spec_queue.complete_item() } } impl Clone for RunContext { fn clone(&self) -> Self { RunContext { stop: self.stop.clone(), read_dir_spec_queue: self.read_dir_spec_queue.clone(), read_dir_result_queue: self.read_dir_result_queue.clone(), core_read_dir_callback: self.core_read_dir_callback.clone(), } } } jwalk-0.8.1/src/lib.rs000064400000000000000000000506631046102023000126570ustar 00000000000000#![warn(clippy::all)] //! Filesystem walk. //! //! - Performed in parallel using rayon //! - Entries streamed in sorted order //! - Custom sort/filter/skip/state //! //! # Example //! //! Recursively iterate over the "foo" directory sorting by name: //! //! ```no_run //! # use std::io::Error; //! use jwalk::{WalkDir}; //! //! # fn try_main() -> Result<(), Error> { //! for entry in WalkDir::new("foo").sort(true) { //! println!("{}", entry?.path().display()); //! } //! # Ok(()) //! # } //! ``` //! # Extended Example //! //! This example uses the //! [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) //! callback for custom: //! 1. **Sort** Entries by name //! 2. **Filter** Errors and hidden files //! 3. **Skip** Content of directories at depth 2 //! 4. **State** Track depth `read_dir_state`. Mark first entry in each //! directory with [`client_state`](struct.DirEntry.html#field.client_state) //! `= true`. //! //! ```no_run //! # use std::io::Error; //! use std::cmp::Ordering; //! use jwalk::{ WalkDirGeneric }; //! //! # fn try_main() -> Result<(), Error> { //! let walk_dir = WalkDirGeneric::<((usize),(bool))>::new("foo") //! .process_read_dir(|depth, path, read_dir_state, children| { //! // 1. Custom sort //! children.sort_by(|a, b| match (a, b) { //! (Ok(a), Ok(b)) => a.file_name.cmp(&b.file_name), //! (Ok(_), Err(_)) => Ordering::Less, //! (Err(_), Ok(_)) => Ordering::Greater, //! (Err(_), Err(_)) => Ordering::Equal, //! }); //! // 2. Custom filter //! children.retain(|dir_entry_result| { //! dir_entry_result.as_ref().map(|dir_entry| { //! dir_entry.file_name //! .to_str() //! .map(|s| s.starts_with('.')) //! .unwrap_or(false) //! }).unwrap_or(false) //! }); //! // 3. Custom skip //! children.iter_mut().for_each(|dir_entry_result| { //! if let Ok(dir_entry) = dir_entry_result { //! if dir_entry.depth == 2 { //! dir_entry.read_children_path = None; //! } //! } //! }); //! // 4. Custom state //! *read_dir_state += 1; //! children.first_mut().map(|dir_entry_result| { //! if let Ok(dir_entry) = dir_entry_result { //! dir_entry.client_state = true; //! } //! }); //! }); //! //! for entry in walk_dir { //! println!("{}", entry?.path().display()); //! } //! # Ok(()) //! # } //! ``` //! # Inspiration //! //! This crate is inspired by both [`walkdir`](https://crates.io/crates/walkdir) //! and [`ignore`](https://crates.io/crates/ignore). It attempts to combine the //! parallelism of `ignore` with `walkdir`'s streaming iterator API. Some code, //! comments, and test are copied directly from `walkdir`. //! //! # Implementation //! //! The following structures are central to the implementation: //! //! ## `ReadDirSpec` //! //! Specification of a future `read_dir` operation. These are stored in the //! `read_dir_spec_queue` in depth first order. When a rayon thread is ready for //! work it pulls the first availible `ReadDirSpec` from this queue. //! //! ## `ReadDir` //! //! Result of a `read_dir` operation generated by rayon thread. These results //! are stored in the `read_dir_result_queue`, also depth first ordered. //! //! ## `ReadDirIter` //! //! Pulls `ReadDir` results from the `read_dir_result_queue`. This iterator is //! driven by calling thread. Results are returned in strict depth first order. //! //! ## `DirEntryIter` //! //! Wraps a `ReadDirIter` and yields individual `DirEntry` results in strict //! depth first order. mod core; use rayon::{ThreadPool, ThreadPoolBuilder}; use std::cmp::Ordering; use std::default::Default; use std::ffi::OsStr; use std::fmt::Debug; use std::fs; use std::path::{Path, PathBuf}; use std::sync::Arc; use crate::core::{ReadDir, ReadDirSpec}; pub use crate::core::{DirEntry, DirEntryIter, Error}; pub use rayon; /// Builder for walking a directory. pub type WalkDir = WalkDirGeneric<((), ())>; /// A specialized Result type for WalkDir. pub type Result = std::result::Result; /// Client state maintained while performing walk. /// /// for state stored in DirEntry's /// [`client_state`](struct.DirEntry.html#field.client_state) field. /// /// Client state can be stored from within the /// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) callback. /// The type of ClientState is determined by WalkDirGeneric type parameter. pub trait ClientState: Send + Default + Debug + 'static { type ReadDirState: Clone + Send + Default + Debug + 'static; type DirEntryState: Send + Default + Debug + 'static; } /// Generic builder for walking a directory. /// /// [`ClientState`](trait.ClientState.html) type parameter allows you to specify /// state to be stored with each DirEntry from within the /// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) /// callback. /// /// Use [`WalkDir`](type.WalkDir.html) if you don't need to store client state /// into yeilded DirEntries. pub struct WalkDirGeneric { root: PathBuf, options: WalkDirOptions, } type ProcessReadDirFunction = dyn Fn(Option, &Path, &mut ::ReadDirState, &mut Vec>>) + Send + Sync + 'static; /// Degree of parallelism to use when performing walk. /// /// Parallelism happens at the directory level. It will help when walking deep /// filesystems with many directories. It wont help when reading a single /// directory with many files. /// /// If you plan to perform lots of per file processing you might want to use Rayon to #[derive(Clone)] pub enum Parallelism { /// Run on calling thread, similar to what happens in the `walkdir` crate. Serial, /// Run in default rayon thread pool. RayonDefaultPool { /// Define when we consider the rayon default pool too busy to serve our iteration and abort the iteration, defaulting to 1s. /// /// This can happen if `jwalk` is launched from within a par-iter on a pool that only has a single thread, /// or if there are many parallel `jwalk` invocations that all use the same threadpool, rendering it too busy /// to respond within this duration. busy_timeout: std::time::Duration, }, /// Run in existing rayon thread pool RayonExistingPool { /// The pool to spawn our work onto. pool: Arc, /// Similar to [`Parallelism::RayonDefaultPool::busy_timeout`] if `Some`, but can be `None` to skip the deadlock check /// in case you know that there is at least one free thread available on the pool. busy_timeout: Option, }, /// Run in new rayon thread pool with # threads RayonNewPool(usize), } struct WalkDirOptions { sort: bool, min_depth: usize, max_depth: usize, skip_hidden: bool, follow_links: bool, parallelism: Parallelism, root_read_dir_state: C::ReadDirState, process_read_dir: Option>>, } impl WalkDirGeneric { /// Create a builder for a recursive directory iterator starting at the file /// path root. If root is a directory, then it is the first item yielded by /// the iterator. If root is a file, then it is the first and only item /// yielded by the iterator. /// /// Note that his iterator can fail on the first element if `into_iter()` is used as it /// has to be infallible. Use [`try_into_iter()`][WalkDirGeneric::try_into_iter()] /// instead for error handling. pub fn new>(root: P) -> Self { WalkDirGeneric { root: root.as_ref().to_path_buf(), options: WalkDirOptions { sort: false, min_depth: 0, max_depth: ::std::usize::MAX, skip_hidden: true, follow_links: false, parallelism: Parallelism::RayonDefaultPool { busy_timeout: std::time::Duration::from_secs(1), }, root_read_dir_state: C::ReadDirState::default(), process_read_dir: None, }, } } /// Try to create an iterator or fail if the rayon threadpool (in any configuration) is busy. pub fn try_into_iter(self) -> Result> { let iter = self.into_iter(); if iter.read_dir_iter.is_none() { Err(Error::busy()) } else { Ok(iter) } } /// Root path of the walk. pub fn root(&self) -> &Path { &self.root } /// Sort entries by `file_name` per directory. Defaults to `false`. Use /// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) for custom /// sorting or filtering. pub fn sort(mut self, sort: bool) -> Self { self.options.sort = sort; self } /// Skip hidden entries. Enabled by default. pub fn skip_hidden(mut self, skip_hidden: bool) -> Self { self.options.skip_hidden = skip_hidden; self } /// Follow symbolic links. By default, this is disabled. /// /// When `yes` is `true`, symbolic links are followed as if they were normal /// directories and files. If a symbolic link is broken or is involved in a /// loop, an error is yielded. /// /// When enabled, the yielded [`DirEntry`] values represent the target of /// the link while the path corresponds to the link. See the [`DirEntry`] /// type for more details. /// /// [`DirEntry`]: struct.DirEntry.html pub fn follow_links(mut self, follow_links: bool) -> Self { self.options.follow_links = follow_links; self } /// Set the minimum depth of entries yielded by the iterator. /// /// The smallest depth is `0` and always corresponds to the path given /// to the `new` function on this type. Its direct descendents have depth /// `1`, and their descendents have depth `2`, and so on. pub fn min_depth(mut self, depth: usize) -> Self { self.options.min_depth = depth; if self.options.min_depth > self.options.max_depth { self.options.min_depth = self.options.max_depth; } self } /// Set the maximum depth of entries yield by the iterator. /// /// The smallest depth is `0` and always corresponds to the path given /// to the `new` function on this type. Its direct descendents have depth /// `1`, and their descendents have depth `2`, and so on. /// /// A depth < 2 will automatically change `parallelism` to /// `Parallelism::Serial`. Parrallelism happens at the `fs::read_dir` level. /// It only makes sense to use multiple threads when reading more then one /// directory. /// /// Note that this will not simply filter the entries of the iterator, but /// it will actually avoid descending into directories when the depth is /// exceeded. pub fn max_depth(mut self, depth: usize) -> Self { self.options.max_depth = depth; if self.options.max_depth < self.options.min_depth { self.options.max_depth = self.options.min_depth; } if self.options.max_depth < 2 { self.options.parallelism = Parallelism::Serial; } self } /// Degree of parallelism to use when performing walk. Defaults to /// [`Parallelism::RayonDefaultPool`](enum.Parallelism.html#variant.RayonDefaultPool). pub fn parallelism(mut self, parallelism: Parallelism) -> Self { self.options.parallelism = parallelism; self } /// Initial ClientState::ReadDirState that is passed to /// [`process_read_dir`](struct.WalkDirGeneric.html#method.process_read_dir) /// when processing root. Defaults to ClientState::ReadDirState::default(). pub fn root_read_dir_state(mut self, read_dir_state: C::ReadDirState) -> Self { self.options.root_read_dir_state = read_dir_state; self } /// A callback function to process (sort/filter/skip/state) each directory /// of entries before they are yielded. Modify the given array to /// sort/filter entries. Use [`entry.read_children_path = /// None`](struct.DirEntry.html#field.read_children_path) to yield a /// directory entry but skip reading its contents. Use /// [`entry.client_state`](struct.DirEntry.html#field.client_state) /// to store custom state with an entry. pub fn process_read_dir(mut self, process_by: F) -> Self where F: Fn(Option, &Path, &mut C::ReadDirState, &mut Vec>>) + Send + Sync + 'static, { self.options.process_read_dir = Some(Arc::new(process_by)); self } } fn process_dir_entry_result( dir_entry_result: Result>, follow_links: bool, ) -> Result> { match dir_entry_result { Ok(mut dir_entry) => { if follow_links && dir_entry.file_type.is_symlink() { dir_entry = dir_entry.follow_symlink()?; } if dir_entry.depth == 0 && dir_entry.file_type.is_symlink() { // As a special case, if we are processing a root entry, then we // always follow it even if it's a symlink and follow_links is // false. We are careful to not let this change the semantics of // the DirEntry however. Namely, the DirEntry should still // respect the follow_links setting. When it's disabled, it // should report itself as a symlink. When it's enabled, it // should always report itself as the target. let metadata = fs::metadata(dir_entry.path()) .map_err(|err| Error::from_path(0, dir_entry.path(), err))?; if metadata.file_type().is_dir() { dir_entry.read_children_path = Some(Arc::from(dir_entry.path())); } } Ok(dir_entry) } Err(err) => Err(err), } } impl IntoIterator for WalkDirGeneric { type Item = Result>; type IntoIter = DirEntryIter; fn into_iter(self) -> DirEntryIter { let sort = self.options.sort; let max_depth = self.options.max_depth; let min_depth = self.options.min_depth; let parallelism = self.options.parallelism; let skip_hidden = self.options.skip_hidden; let follow_links = self.options.follow_links; let process_read_dir = self.options.process_read_dir.clone(); let mut root_read_dir_state = self.options.root_read_dir_state; let follow_link_ancestors = if follow_links { Arc::new(vec![Arc::from(self.root.clone()) as Arc]) } else { Arc::new(vec![]) }; let root_entry = DirEntry::from_path(0, &self.root, false, follow_link_ancestors); let root_parent_path = root_entry .as_ref() .map(|root| root.parent_path().to_owned()) .unwrap_or_default(); let mut root_entry_results = vec![process_dir_entry_result(root_entry, follow_links)]; if let Some(process_read_dir) = process_read_dir.as_ref() { process_read_dir( None, &root_parent_path, &mut root_read_dir_state, &mut root_entry_results, ); } DirEntryIter::new( root_entry_results, parallelism, min_depth, root_read_dir_state, Arc::new(move |read_dir_spec| { let ReadDirSpec { path, depth, mut client_read_state, mut follow_link_ancestors, } = read_dir_spec; let read_dir_depth = depth; let read_dir_contents_depth = depth + 1; if read_dir_contents_depth > max_depth { return Ok(ReadDir::new(client_read_state, Vec::new())); } follow_link_ancestors = if follow_links { let mut ancestors = Vec::with_capacity(follow_link_ancestors.len() + 1); ancestors.extend(follow_link_ancestors.iter().cloned()); ancestors.push(path.clone()); Arc::new(ancestors) } else { follow_link_ancestors }; let mut dir_entry_results: Vec<_> = fs::read_dir(path.as_ref()) .map_err(|err| Error::from_path(0, path.to_path_buf(), err))? .filter_map(|dir_entry_result| { let fs_dir_entry = match dir_entry_result { Ok(fs_dir_entry) => fs_dir_entry, Err(err) => { return Some(Err(Error::from_io(read_dir_contents_depth, err))) } }; let dir_entry = match DirEntry::from_entry( read_dir_contents_depth, path.clone(), &fs_dir_entry, follow_link_ancestors.clone(), ) { Ok(dir_entry) => dir_entry, Err(err) => return Some(Err(err)), }; if skip_hidden && is_hidden(&dir_entry.file_name) { return None; } Some(process_dir_entry_result(Ok(dir_entry), follow_links)) }) .collect(); if sort { dir_entry_results.sort_by(|a, b| match (a, b) { (Ok(a), Ok(b)) => a.file_name.cmp(&b.file_name), (Ok(_), Err(_)) => Ordering::Less, (Err(_), Ok(_)) => Ordering::Greater, (Err(_), Err(_)) => Ordering::Equal, }); } if let Some(process_read_dir) = process_read_dir.as_ref() { process_read_dir( Some(read_dir_depth), path.as_ref(), &mut client_read_state, &mut dir_entry_results, ); } Ok(ReadDir::new(client_read_state, dir_entry_results)) }), ) } } impl Clone for WalkDirOptions { fn clone(&self) -> WalkDirOptions { WalkDirOptions { sort: false, min_depth: self.min_depth, max_depth: self.max_depth, skip_hidden: self.skip_hidden, follow_links: self.follow_links, parallelism: self.parallelism.clone(), root_read_dir_state: self.root_read_dir_state.clone(), process_read_dir: self.process_read_dir.clone(), } } } impl Parallelism { pub(crate) fn spawn(&self, op: OP) where OP: FnOnce() + Send + 'static, { match self { Parallelism::Serial => op(), Parallelism::RayonDefaultPool { .. } => rayon::spawn(op), Parallelism::RayonNewPool(num_threads) => { let mut thread_pool = ThreadPoolBuilder::new(); if *num_threads > 0 { thread_pool = thread_pool.num_threads(*num_threads); } if let Ok(thread_pool) = thread_pool.build() { thread_pool.spawn(op); } else { rayon::spawn(op); } } Parallelism::RayonExistingPool { pool, .. } => pool.spawn(op), } } pub(crate) fn timeout(&self) -> Option { match self { Parallelism::Serial | Parallelism::RayonNewPool(_) => None, Parallelism::RayonDefaultPool { busy_timeout } => Some(*busy_timeout), Parallelism::RayonExistingPool { busy_timeout, .. } => *busy_timeout, } } } fn is_hidden(file_name: &OsStr) -> bool { file_name .to_str() .map(|s| s.starts_with('.')) .unwrap_or(false) } impl ClientState for (B, E) where B: Clone + Send + Default + Debug + 'static, E: Send + Default + Debug + 'static, { type ReadDirState = B; type DirEntryState = E; } jwalk-0.8.1/tests/assets/test_dir/a.txt000064400000000000000000000000021046102023000161740ustar 00000000000000a jwalk-0.8.1/tests/assets/test_dir/b.txt000064400000000000000000000000111046102023000161750ustar 00000000000000b find mejwalk-0.8.1/tests/assets/test_dir/c.txt000064400000000000000000000000021046102023000161760ustar 00000000000000c jwalk-0.8.1/tests/assets/test_dir/group 1/d.txt000064400000000000000000000000071046102023000174610ustar 00000000000000find mejwalk-0.8.1/tests/assets/test_dir/group 2/.hidden_file.txt000064400000000000000000000000001046102023000215400ustar 00000000000000jwalk-0.8.1/tests/assets/test_dir/group 2/e.txt000064400000000000000000000000071046102023000174630ustar 00000000000000find mejwalk-0.8.1/tests/detect_deadlock.rs000064400000000000000000000020671046102023000155550ustar 00000000000000use jwalk::WalkDir; use rayon::prelude::*; #[test] fn works() { rayon::ThreadPoolBuilder::new() .num_threads(1) .build_global() .expect("Failed to initialize worker thread pool"); // Does not finish if jwalk uses shared pool with 1 thread, but we can detect this issue and signal this with an error. (0..=1) .collect::>() .par_iter() .for_each(|round| { let generic = WalkDir::new(".").parallelism(jwalk::Parallelism::RayonDefaultPool { busy_timeout: std::time::Duration::from_millis(10), }); if *round == 0 { for entry in generic { match entry { Ok(_) => panic!("Must detect deadlock"), Err(err) if err.is_busy() => {} Err(err) => panic!("Unexpected error: {:?}", err), } } } else { assert!(matches!(generic.try_into_iter(), Err(err) if err.is_busy())); } }); } jwalk-0.8.1/tests/integration.rs000064400000000000000000000743621046102023000150110ustar 00000000000000use lazy_static::lazy_static; use rayon::iter::ParallelIterator; use rayon::prelude::*; use std::env; use std::fs; use std::path::PathBuf; use std::sync::Mutex; mod util; use jwalk::*; use util::Dir; #[test] fn empty() { let dir = Dir::tmp(); let wd = WalkDir::new(dir.path()); let r = dir.run_recursive(wd); r.assert_no_errors(); assert_eq!(1, r.ents().len()); let ent = &r.ents()[0]; assert!(ent.file_type().is_dir()); assert!(!ent.path_is_symlink()); assert_eq!(0, ent.depth()); assert_eq!(dir.path(), ent.path()); assert_eq!(dir.path().file_name().unwrap(), ent.file_name()); } #[test] fn empty_follow() { let dir = Dir::tmp(); let wd = WalkDir::new(dir.path()).follow_links(true); let r = dir.run_recursive(wd); r.assert_no_errors(); assert_eq!(1, r.ents().len()); let ent = &r.ents()[0]; assert!(ent.file_type().is_dir()); assert!(!ent.path_is_symlink()); assert_eq!(0, ent.depth()); assert_eq!(dir.path(), ent.path()); assert_eq!(dir.path().file_name().unwrap(), ent.file_name()); } #[test] fn empty_file() { let dir = Dir::tmp(); dir.touch("a"); let wd = WalkDir::new(dir.path().join("a")); let r = dir.run_recursive(wd); r.assert_no_errors(); assert_eq!(1, r.ents().len()); let ent = &r.ents()[0]; assert!(ent.file_type().is_file()); assert!(!ent.path_is_symlink()); assert_eq!(0, ent.depth()); assert_eq!(dir.join("a"), ent.path()); assert_eq!("a", ent.file_name()); } #[test] fn empty_file_follow() { let dir = Dir::tmp(); dir.touch("a"); let wd = WalkDir::new(dir.path().join("a")).follow_links(true); let r = dir.run_recursive(wd); r.assert_no_errors(); assert_eq!(1, r.ents().len()); let ent = &r.ents()[0]; assert!(ent.file_type().is_file()); assert!(!ent.path_is_symlink()); assert_eq!(0, ent.depth()); assert_eq!(dir.join("a"), ent.path()); assert_eq!("a", ent.file_name()); } #[test] fn one_dir() { let dir = Dir::tmp(); dir.mkdirp("a"); let wd = WalkDir::new(dir.path()); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(2, ents.len()); let ent = &ents[1]; assert_eq!(dir.join("a"), ent.path()); assert_eq!(1, ent.depth()); assert_eq!("a", ent.file_name()); assert!(ent.file_type().is_dir()); } #[test] fn one_file() { let dir = Dir::tmp(); dir.touch("a"); let wd = WalkDir::new(dir.path()); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(2, ents.len()); let ent = &ents[1]; assert_eq!(dir.join("a"), ent.path()); assert_eq!(1, ent.depth()); assert_eq!("a", ent.file_name()); assert!(ent.file_type().is_file()); } #[test] fn one_dir_one_file() { let dir = Dir::tmp(); dir.mkdirp("foo"); dir.touch("foo/a"); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("foo"), dir.join("foo").join("a"), ]; assert_eq!(expected, r.paths()); } #[test] fn many_files() { let dir = Dir::tmp(); dir.mkdirp("foo"); dir.touch_all(&["foo/a", "foo/b", "foo/c"]); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("foo"), dir.join("foo").join("a"), dir.join("foo").join("b"), dir.join("foo").join("c"), ]; assert_eq!(expected, r.paths()); } #[test] fn many_dirs() { let dir = Dir::tmp(); dir.mkdirp("foo/a"); dir.mkdirp("foo/b"); dir.mkdirp("foo/c"); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("foo"), dir.join("foo").join("a"), dir.join("foo").join("b"), dir.join("foo").join("c"), ]; assert_eq!(expected, r.paths()); } #[test] fn many_mixed() { let dir = Dir::tmp(); dir.mkdirp("foo/a"); dir.mkdirp("foo/c"); dir.mkdirp("foo/e"); dir.touch_all(&["foo/b", "foo/d", "foo/f"]); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("foo"), dir.join("foo").join("a"), dir.join("foo").join("b"), dir.join("foo").join("c"), dir.join("foo").join("d"), dir.join("foo").join("e"), dir.join("foo").join("f"), ]; assert_eq!(expected, r.paths()); } #[test] fn nested() { let nested = PathBuf::from("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z"); let dir = Dir::tmp(); dir.mkdirp(&nested); dir.touch(nested.join("A")); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("a"), dir.join("a/b"), dir.join("a/b/c"), dir.join("a/b/c/d"), dir.join("a/b/c/d/e"), dir.join("a/b/c/d/e/f"), dir.join("a/b/c/d/e/f/g"), dir.join("a/b/c/d/e/f/g/h"), dir.join("a/b/c/d/e/f/g/h/i"), dir.join("a/b/c/d/e/f/g/h/i/j"), dir.join("a/b/c/d/e/f/g/h/i/j/k"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y"), dir.join("a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q/r/s/t/u/v/w/x/y/z"), dir.join(&nested).join("A"), ]; assert_eq!(expected, r.paths()); } #[test] fn siblings() { let dir = Dir::tmp(); dir.mkdirp("foo"); dir.mkdirp("bar"); dir.touch_all(&["foo/a", "foo/b"]); dir.touch_all(&["bar/a", "bar/b"]); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("bar"), dir.join("bar").join("a"), dir.join("bar").join("b"), dir.join("foo"), dir.join("foo").join("a"), dir.join("foo").join("b"), ]; assert_eq!(expected, r.paths()); } #[test] fn sym_root_file_nofollow() { let dir = Dir::tmp(); dir.touch("a"); dir.symlink_file("a", "a-link"); let wd = WalkDir::new(dir.join("a-link")).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(1, ents.len()); let link = &ents[0]; assert_eq!(dir.join("a-link"), link.path()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(0, link.depth()); assert!(link.file_type().is_symlink()); assert!(!link.file_type().is_file()); assert!(!link.file_type().is_dir()); assert!(link.metadata().unwrap().file_type().is_symlink()); assert!(!link.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().is_dir()); } #[test] fn sym_root_file_follow() { let dir = Dir::tmp(); dir.touch("a"); dir.symlink_file("a", "a-link"); let wd = WalkDir::new(dir.join("a-link")) .sort(true) .follow_links(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); let link = &ents[0]; assert_eq!(dir.join("a-link"), link.path()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(0, link.depth()); assert!(!link.file_type().is_symlink()); assert!(link.file_type().is_file()); assert!(!link.file_type().is_dir()); assert!(!link.metadata().unwrap().file_type().is_symlink()); assert!(link.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().is_dir()); } #[test] fn sym_root_dir_nofollow() { let dir = Dir::tmp(); dir.mkdirp("a"); dir.symlink_dir("a", "a-link"); dir.touch("a/zzz"); let wd = WalkDir::new(dir.join("a-link")).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(2, ents.len()); let link = &ents[0]; assert_eq!(dir.join("a-link"), link.path()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(0, link.depth()); assert!(link.file_type().is_symlink()); assert!(!link.file_type().is_file()); assert!(!link.file_type().is_dir()); assert!(link.metadata().unwrap().file_type().is_symlink()); assert!(!link.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().is_dir()); let link_zzz = &ents[1]; assert_eq!(dir.join("a-link").join("zzz"), link_zzz.path()); assert!(!link_zzz.path_is_symlink()); } #[test] fn sym_root_dir_follow() { let dir = Dir::tmp(); dir.mkdirp("a"); dir.symlink_dir("a", "a-link"); dir.touch("a/zzz"); let wd = WalkDir::new(dir.join("a-link")) .sort(true) .follow_links(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(2, ents.len()); let link = &ents[0]; assert_eq!(dir.join("a-link"), link.path()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(0, link.depth()); assert!(!link.file_type().is_symlink()); assert!(!link.file_type().is_file()); assert!(link.file_type().is_dir()); assert!(!link.metadata().unwrap().file_type().is_symlink()); assert!(!link.metadata().unwrap().is_file()); assert!(link.metadata().unwrap().is_dir()); let link_zzz = &ents[1]; assert_eq!(dir.join("a-link").join("zzz"), link_zzz.path()); assert!(!link_zzz.path_is_symlink()); } #[test] fn sym_file_nofollow() { let dir = Dir::tmp(); dir.touch("a"); dir.symlink_file("a", "a-link"); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(3, ents.len()); let (src, link) = (&ents[1], &ents[2]); assert_eq!(dir.join("a"), src.path()); assert_eq!(dir.join("a-link"), link.path()); assert!(!src.path_is_symlink()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(1, src.depth()); assert_eq!(1, link.depth()); assert!(src.file_type().is_file()); assert!(link.file_type().is_symlink()); assert!(!link.file_type().is_file()); assert!(!link.file_type().is_dir()); assert!(src.metadata().unwrap().is_file()); assert!(link.metadata().unwrap().file_type().is_symlink()); assert!(!link.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().is_dir()); } #[test] fn sym_file_follow() { let dir = Dir::tmp(); dir.touch("a"); dir.symlink_file("a", "a-link"); let wd = WalkDir::new(dir.path()).sort(true).follow_links(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(3, ents.len()); let (src, link) = (&ents[1], &ents[2]); assert_eq!(dir.join("a"), src.path()); assert_eq!(dir.join("a-link"), link.path()); assert!(!src.path_is_symlink()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(1, src.depth()); assert_eq!(1, link.depth()); assert!(src.file_type().is_file()); assert!(!link.file_type().is_symlink()); assert!(link.file_type().is_file()); assert!(!link.file_type().is_dir()); assert!(src.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().file_type().is_symlink()); assert!(link.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().is_dir()); } #[test] fn sym_dir_nofollow() { let dir = Dir::tmp(); dir.mkdirp("a"); dir.symlink_dir("a", "a-link"); dir.touch("a/zzz"); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(4, ents.len()); let (src, link) = (&ents[1], &ents[3]); assert_eq!(dir.join("a"), src.path()); assert_eq!(dir.join("a-link"), link.path()); assert!(!src.path_is_symlink()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(1, src.depth()); assert_eq!(1, link.depth()); assert!(src.file_type().is_dir()); assert!(link.file_type().is_symlink()); assert!(!link.file_type().is_file()); assert!(!link.file_type().is_dir()); assert!(src.metadata().unwrap().is_dir()); assert!(link.metadata().unwrap().file_type().is_symlink()); assert!(!link.metadata().unwrap().is_file()); assert!(!link.metadata().unwrap().is_dir()); } #[test] fn sym_dir_follow() { let dir = Dir::tmp(); dir.mkdirp("a"); dir.symlink_dir("a", "a-link"); dir.touch("a/zzz"); let wd = WalkDir::new(dir.path()).follow_links(true).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let ents = r.ents(); assert_eq!(5, ents.len()); let (src, link) = (&ents[1], &ents[3]); assert_eq!(dir.join("a"), src.path()); assert_eq!(dir.join("a-link"), link.path()); assert!(!src.path_is_symlink()); assert!(link.path_is_symlink()); assert_eq!(dir.join("a"), fs::read_link(link.path()).unwrap()); assert_eq!(1, src.depth()); assert_eq!(1, link.depth()); assert!(src.file_type().is_dir()); assert!(!link.file_type().is_symlink()); assert!(!link.file_type().is_file()); assert!(link.file_type().is_dir()); assert!(src.metadata().unwrap().is_dir()); assert!(!link.metadata().unwrap().file_type().is_symlink()); assert!(!link.metadata().unwrap().is_file()); assert!(link.metadata().unwrap().is_dir()); let (src_zzz, link_zzz) = (&ents[2], &ents[4]); assert_eq!(dir.join("a").join("zzz"), src_zzz.path()); assert_eq!(dir.join("a-link").join("zzz"), link_zzz.path()); assert!(!src_zzz.path_is_symlink()); assert!(!link_zzz.path_is_symlink()); } #[test] fn sym_noloop() { let dir = Dir::tmp(); dir.mkdirp("a/b/c"); dir.symlink_dir("a", "a/b/c/a-link"); let wd = WalkDir::new(dir.path()); let r = dir.run_recursive(wd); // There's no loop if we aren't following symlinks. r.assert_no_errors(); assert_eq!(5, r.ents().len()); } #[test] fn sym_loop_detect() { let dir = Dir::tmp(); dir.mkdirp("a/b/c"); dir.symlink_dir("a", "a/b/c/a-link"); let wd = WalkDir::new(dir.path()).follow_links(true); let r = dir.run_recursive(wd); let (ents, errs) = (r.ents(), r.errs()); assert_eq!(4, ents.len()); assert_eq!(1, errs.len()); let err = &errs[0]; let expected = dir.join("a/b/c/a-link"); assert_eq!(Some(&*expected), err.path()); let expected = dir.join("a"); assert_eq!(Some(&*expected), err.loop_ancestor()); assert_eq!(4, err.depth()); assert!(err.io_error().is_none()); } #[test] fn sym_self_loop_no_error() { let dir = Dir::tmp(); dir.symlink_file("a", "a"); let wd = WalkDir::new(dir.path()); let r = dir.run_recursive(wd); // No errors occur because even though the symlink points to nowhere, it // is never followed, and thus no error occurs. r.assert_no_errors(); assert_eq!(2, r.ents().len()); let ent = &r.ents()[1]; assert_eq!(dir.join("a"), ent.path()); assert!(ent.path_is_symlink()); assert!(ent.file_type().is_symlink()); assert!(!ent.file_type().is_file()); assert!(!ent.file_type().is_dir()); assert!(ent.metadata().unwrap().file_type().is_symlink()); assert!(!ent.metadata().unwrap().file_type().is_file()); assert!(!ent.metadata().unwrap().file_type().is_dir()); } #[test] fn sym_file_self_loop_io_error() { let dir = Dir::tmp(); dir.symlink_file("a", "a"); let wd = WalkDir::new(dir.path()).follow_links(true); let r = dir.run_recursive(wd); let (ents, errs) = (r.ents(), r.errs()); assert_eq!(1, ents.len()); assert_eq!(1, errs.len()); let err = &errs[0]; let expected = dir.join("a"); assert_eq!(Some(&*expected), err.path()); assert_eq!(1, err.depth()); assert!(err.loop_ancestor().is_none()); assert!(err.io_error().is_some()); } #[test] fn sym_dir_self_loop_io_error() { let dir = Dir::tmp(); dir.symlink_dir("a", "a"); let wd = WalkDir::new(dir.path()).follow_links(true); let r = dir.run_recursive(wd); let (ents, errs) = (r.ents(), r.errs()); assert_eq!(1, ents.len()); assert_eq!(1, errs.len()); let err = &errs[0]; let expected = dir.join("a"); assert_eq!(Some(&*expected), err.path()); assert_eq!(1, err.depth()); assert!(err.loop_ancestor().is_none()); assert!(err.io_error().is_some()); } #[test] fn min_depth_1() { let dir = Dir::tmp(); dir.mkdirp("a/b"); let wd = WalkDir::new(dir.path()).min_depth(1).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![dir.join("a"), dir.join("a").join("b")]; assert_eq!(expected, r.paths()); } #[test] fn min_depth_2() { let dir = Dir::tmp(); dir.mkdirp("a/b"); let wd = WalkDir::new(dir.path()).min_depth(2).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![dir.join("a").join("b")]; assert_eq!(expected, r.paths()); } #[test] fn max_depth_0() { let dir = Dir::tmp(); dir.mkdirp("a/b"); let wd = WalkDir::new(dir.path()).max_depth(0).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![dir.path().to_path_buf()]; assert_eq!(expected, r.paths()); } #[test] fn max_depth_1() { let dir = Dir::tmp(); dir.mkdirp("a/b"); let wd = WalkDir::new(dir.path()).max_depth(1).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![dir.path().to_path_buf(), dir.join("a")]; assert_eq!(expected, r.paths()); } #[test] fn max_depth_2() { let dir = Dir::tmp(); dir.mkdirp("a/b"); let wd = WalkDir::new(dir.path()).max_depth(2).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("a"), dir.join("a").join("b"), ]; assert_eq!(expected, r.paths()); } #[test] fn min_max_depth_diff_0() { let dir = Dir::tmp(); dir.mkdirp("a/b/c"); let wd = WalkDir::new(dir.path()) .min_depth(2) .max_depth(2) .sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![dir.join("a").join("b")]; assert_eq!(expected, r.paths()); } #[test] fn min_max_depth_diff_1() { let dir = Dir::tmp(); dir.mkdirp("a/b/c"); let wd = WalkDir::new(dir.path()) .min_depth(1) .max_depth(2) .sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![dir.join("a"), dir.join("a").join("b")]; assert_eq!(expected, r.paths()); } #[test] fn sort() { let dir = Dir::tmp(); dir.mkdirp("foo/bar/baz/abc"); dir.mkdirp("quux"); let wd = WalkDir::new(dir.path()).sort(true); let r = dir.run_recursive(wd); r.assert_no_errors(); let expected = vec![ dir.path().to_path_buf(), dir.join("foo"), dir.join("foo").join("bar"), dir.join("foo").join("bar").join("baz"), dir.join("foo").join("bar").join("baz").join("abc"), dir.join("quux"), ]; assert_eq!(expected, r.paths()); } fn test_dir() -> (PathBuf, tempfile::TempDir) { let template = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("tests/assets/test_dir"); let temp_dir = tempfile::tempdir().unwrap(); let options = fs_extra::dir::CopyOptions::new(); fs_extra::dir::copy(&template, &temp_dir, &options).unwrap(); let mut test_dir = temp_dir.path().to_path_buf(); test_dir.push(template.file_name().unwrap()); (test_dir, temp_dir) } fn local_paths(walk_dir: WalkDir) -> Vec { let root = walk_dir.root().to_owned(); walk_dir .into_iter() .map(|each_result| { let each_entry = each_result.unwrap(); if let Some(err) = each_entry.read_children_error.as_ref() { panic!("should not encounter any child errors :{:?}", err); } let path = each_entry.path(); let path = path.strip_prefix(&root).unwrap().to_path_buf(); let mut path_string = path.to_str().unwrap().to_string(); path_string.push_str(&format!(" ({})", each_entry.depth)); path_string }) .collect() } #[test] fn walk_serial() { let (test_dir, _temp_dir) = test_dir(); let paths = local_paths( WalkDir::new(test_dir) .parallelism(Parallelism::Serial) .sort(true), ); assert_eq!( paths, vec![ " (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)", "group 1 (1)", "group 1/d.txt (2)", "group 2 (1)", "group 2/e.txt (2)", ] ); } #[test] fn sort_by_name_rayon_custom_2_threads() { let (test_dir, _temp_dir) = test_dir(); let paths = local_paths( WalkDir::new(test_dir) .parallelism(Parallelism::RayonNewPool(2)) .sort(true), ); assert_eq!( paths, vec![ " (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)", "group 1 (1)", "group 1/d.txt (2)", "group 2 (1)", "group 2/e.txt (2)", ] ); } #[test] fn walk_rayon_global() { let (test_dir, _temp_dir) = test_dir(); let paths = local_paths(WalkDir::new(test_dir).sort(true)); assert_eq!( paths, vec![ " (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)", "group 1 (1)", "group 1/d.txt (2)", "group 2 (1)", "group 2/e.txt (2)", ] ); } #[test] fn walk_rayon_no_lockup() { // Without jwalk_par_bridge this locks (pre rayon 1.6.1) // This test now passes without needing jwalk_par_bridge // and that code has been removed from jwalk. let pool = std::sync::Arc::new( rayon::ThreadPoolBuilder::new() .num_threads(1) .build() .unwrap(), ); let _: Vec<_> = WalkDir::new(PathBuf::from(env!("CARGO_MANIFEST_DIR"))) .parallelism(Parallelism::RayonExistingPool { pool, busy_timeout: std::time::Duration::from_millis(500).into(), }) .process_read_dir(|_, _, _, dir_entry_results| { for dir_entry_result in dir_entry_results { let _ = dir_entry_result .as_ref() .map(|dir_entry| dir_entry.metadata()); } }) .sort(true) .into_iter() .collect(); } #[test] fn combine_with_rayon_no_lockup_1() { // only run this test if linux_checkout present let linux_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("benches/assets/linux_checkout"); if linux_dir.exists() { rayon::scope(|_| { eprintln!("WalkDir…"); for _entry in WalkDir::new(linux_dir) {} eprintln!("WalkDir completed"); }); } } #[test] fn combine_with_rayon_no_lockup_2() { WalkDir::new(PathBuf::from(env!("CARGO_MANIFEST_DIR"))) .sort(true) .into_iter() .par_bridge() .filter_map(|dir_entry_result| { let dir_entry = dir_entry_result.ok()?; if dir_entry.file_type().is_file() { let path = dir_entry.path(); let text = std::fs::read_to_string(path).ok()?; if text.contains("hello world") { return Some(true); } } None }) .count(); } #[test] fn see_hidden_files() { let (test_dir, _temp_dir) = test_dir(); let paths = local_paths(WalkDir::new(test_dir).skip_hidden(false).sort(true)); assert!(paths.contains(&"group 2/.hidden_file.txt (2)".to_string())); } #[test] fn walk_file() { let (test_dir, _temp_dir) = test_dir(); let walk_dir = WalkDir::new(test_dir.join("a.txt")); let mut iter = walk_dir.into_iter(); assert_eq!( iter.next().unwrap().unwrap().file_name.to_str().unwrap(), "a.txt" ); assert!(iter.next().is_none()); } #[test] fn walk_file_serial() { let (test_dir, _temp_dir) = test_dir(); let walk_dir = WalkDir::new(test_dir.join("a.txt")).parallelism(Parallelism::Serial); let mut iter = walk_dir.into_iter(); assert_eq!( iter.next().unwrap().unwrap().file_name.to_str().unwrap(), "a.txt" ); assert!(iter.next().is_none()); } #[test] fn error_when_path_does_not_exist() { let (test_dir, _temp_dir) = test_dir(); let walk_dir = WalkDir::new(test_dir.join("path_does_not_exist")); let mut iter = walk_dir.into_iter(); assert!(iter.next().unwrap().is_err()); assert!(iter.next().is_none()); } #[test] fn error_when_path_removed_durring_iteration() { let (test_dir, _temp_dir) = test_dir(); let walk_dir = WalkDir::new(&test_dir) .parallelism(Parallelism::Serial) .sort(true); let mut iter = walk_dir.into_iter(); // Read root. read_dir for root is also called since single thread mode. let _ = iter.next().unwrap().is_ok(); // " (0)", // Remove group 2 dir from disk fs_extra::remove_items(&[test_dir.join("group 2")]).unwrap(); let _ = iter.next().unwrap().is_ok(); // "a.txt (1)", let _ = iter.next().unwrap().is_ok(); // "b.txt (1)", let _ = iter.next().unwrap().is_ok(); // "c.txt (1)", let _ = iter.next().unwrap().is_ok(); // "group 1 (1)", let _ = iter.next().unwrap().is_ok(); // "group 1/d.txt (2)", // group 2 is read correctly, since it was read before path removed. let group_2 = iter.next().unwrap().unwrap(); // group 2 content error IS set, since path is removed when try read_dir for // group 2 path. let _ = group_2.read_children_error.is_some(); // done! assert!(iter.next().is_none()); } #[test] fn walk_root() { let paths: Vec<_> = WalkDir::new("/") .max_depth(1) .sort(true) .into_iter() .filter_map(|each| Some(each.ok()?.path())) .collect(); assert_eq!(paths.first().unwrap().to_str().unwrap(), "/"); } lazy_static! { static ref RELATIVE_MUTEX: Mutex<()> = Mutex::new(()); } #[test] fn walk_relative_1() { let _shared = RELATIVE_MUTEX.lock().unwrap(); let (test_dir, _temp_dir) = test_dir(); env::set_current_dir(&test_dir).unwrap(); let paths = local_paths(WalkDir::new(".").sort(true)); assert_eq!( paths, vec![ " (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)", "group 1 (1)", "group 1/d.txt (2)", "group 2 (1)", "group 2/e.txt (2)", ] ); let root_dir_entry = WalkDir::new("..").into_iter().next().unwrap().unwrap(); assert_eq!(&root_dir_entry.file_name, ".."); } #[test] fn walk_relative_2() { let _shared = RELATIVE_MUTEX.lock().unwrap(); let (test_dir, _temp_dir) = test_dir(); env::set_current_dir(&test_dir.join("group 1")).unwrap(); let paths = local_paths(WalkDir::new("..").sort(true)); assert_eq!( paths, vec![ " (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)", "group 1 (1)", "group 1/d.txt (2)", "group 2 (1)", "group 2/e.txt (2)", ] ); let root_dir_entry = WalkDir::new(".").into_iter().next().unwrap().unwrap(); assert_eq!(&root_dir_entry.file_name, "."); } #[test] fn filter_groups_with_process_read_dir() { let (test_dir, _temp_dir) = test_dir(); let paths = local_paths( WalkDir::new(test_dir) .sort(true) // Filter groups out manually .process_read_dir(|_depth, _path, _parent, children| { children.retain(|each_result| { each_result .as_ref() .map(|dir_entry| { !dir_entry.file_name.to_string_lossy().starts_with("group") }) .unwrap_or(true) }); }), ); assert_eq!(paths, vec![" (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)",]); } #[test] fn filter_group_children_with_process_read_dir() { let (test_dir, _temp_dir) = test_dir(); let paths = local_paths( WalkDir::new(test_dir) .sort(true) // Filter group children .process_read_dir(|_depth, _path, _parent, children| { children.iter_mut().for_each(|each_result| { if let Ok(each) = each_result { if each.file_name.to_string_lossy().starts_with("group") { each.read_children_path = None; } } }); }), ); assert_eq!( paths, vec![ " (0)", "a.txt (1)", "b.txt (1)", "c.txt (1)", "group 1 (1)", "group 2 (1)", ] ); } #[test] fn test_read_linux() { // only run this test if linux_checkout present let linux_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("benches/assets/linux_checkout"); if linux_dir.exists() { for each in WalkDir::new(linux_dir) { let path = each.unwrap().path(); assert!(path.exists(), "{:?}", path); } } } jwalk-0.8.1/tests/util/mod.rs000064400000000000000000000164521046102023000142160ustar 00000000000000use std::env; use std::error; use std::fs::{self, File}; use std::io::{self}; use std::path::{Path, PathBuf}; use std::result; use jwalk::*; /// Create an error from a format!-like syntax. #[macro_export] macro_rules! err { ($($tt:tt)*) => { Box::::from(format!($($tt)*)) } } /// A convenient result type alias. pub type Result = result::Result>; /// The result of running a recursive directory iterator on a single directory. #[derive(Debug)] pub struct RecursiveResults { ents: Vec>, errs: Vec, } impl RecursiveResults { /// Return all of the errors encountered during traversal. pub fn errs(&self) -> &[Error] { &self.errs } /// Assert that no errors have occurred. pub fn assert_no_errors(&self) { assert!( self.errs.is_empty(), "expected to find no errors, but found: {:?}", self.errs ); } /// Return all the successfully retrieved directory entries in the order /// in which they were retrieved. pub fn ents(&self) -> &[DirEntry] { &self.ents } /// Return all paths from all successfully retrieved directory entries. /// /// This does not include paths that correspond to an error. pub fn paths(&self) -> Vec { self.ents.iter().map(|d| d.path()).collect() } /* /// Return all the successfully retrieved directory entries, sorted /// lexicographically by their full file path. pub fn sorted_ents(&self) -> Vec> { let mut ents = self.ents.clone(); ents.sort_by(|e1, e2| e1.path().cmp(e2.path())); ents } /// Return all paths from all successfully retrieved directory entries, /// sorted lexicographically. /// /// This does not include paths that correspond to an error. pub fn sorted_paths(&self) -> Vec { self.sorted_ents().into_iter().map(|d| d.into_path()).collect() }*/ } /// A helper for managing a directory in which to run tests. /// /// When manipulating paths within this directory, paths are interpreted /// relative to this directory. #[derive(Debug)] pub struct Dir { dir: TempDir, } impl Dir { /// Create a new empty temporary directory. pub fn tmp() -> Dir { let dir = TempDir::new().unwrap(); Dir { dir } } /// Return the path to this directory. pub fn path(&self) -> &Path { self.dir.path() } /// Return a path joined to the path to this directory. pub fn join>(&self, path: P) -> PathBuf { self.path().join(path) } /// Run the given iterator and return the result as a distinct collection /// of directory entries and errors. pub fn run_recursive(&self, it: I) -> RecursiveResults where C: ClientState, I: IntoIterator, Error>>, { let mut results = RecursiveResults { ents: vec![], errs: vec![], }; for result in it { match result { Ok(ent) => results.ents.push(ent), Err(err) => results.errs.push(err), } } results } /// Create a directory at the given path, while creating all intermediate /// directories as needed. pub fn mkdirp>(&self, path: P) { let full = self.join(path); fs::create_dir_all(&full) .map_err(|e| err!("failed to create directory {}: {}", full.display(), e)) .unwrap(); } /// Create an empty file at the given path. All ancestor directories must /// already exists. pub fn touch>(&self, path: P) { let full = self.join(path); File::create(&full) .map_err(|e| err!("failed to create file {}: {}", full.display(), e)) .unwrap(); } /// Create empty files at the given paths. All ancestor directories must /// already exists. pub fn touch_all>(&self, paths: &[P]) { for p in paths { self.touch(p); } } /// Create a file symlink to the given src with the given link name. pub fn symlink_file, P2: AsRef>(&self, src: P1, link_name: P2) { #[cfg(windows)] fn imp(src: &Path, link_name: &Path) -> io::Result<()> { use std::os::windows::fs::symlink_file; symlink_file(src, link_name) } #[cfg(unix)] fn imp(src: &Path, link_name: &Path) -> io::Result<()> { use std::os::unix::fs::symlink; symlink(src, link_name) } let (src, link_name) = (self.join(src), self.join(link_name)); imp(&src, &link_name) .map_err(|e| { err!( "failed to symlink file {} with target {}: {}", src.display(), link_name.display(), e ) }) .unwrap() } /// Create a directory symlink to the given src with the given link name. pub fn symlink_dir, P2: AsRef>(&self, src: P1, link_name: P2) { #[cfg(windows)] fn imp(src: &Path, link_name: &Path) -> io::Result<()> { use std::os::windows::fs::symlink_dir; symlink_dir(src, link_name) } #[cfg(unix)] fn imp(src: &Path, link_name: &Path) -> io::Result<()> { use std::os::unix::fs::symlink; symlink(src, link_name) } let (src, link_name) = (self.join(src), self.join(link_name)); imp(&src, &link_name) .map_err(|e| { err!( "failed to symlink directory {} with target {}: {}", src.display(), link_name.display(), e ) }) .unwrap() } } /// A simple wrapper for creating a temporary directory that is automatically /// deleted when it's dropped. /// /// We use this in lieu of tempfile because tempfile brings in too many /// dependencies. #[derive(Debug)] pub struct TempDir(PathBuf); impl Drop for TempDir { fn drop(&mut self) { fs::remove_dir_all(&self.0).unwrap(); } } impl TempDir { /// Create a new empty temporary directory under the system's configured /// temporary directory. pub fn new() -> Result { #[allow(deprecated)] use std::sync::atomic::{AtomicUsize, Ordering, ATOMIC_USIZE_INIT}; static TRIES: usize = 100; #[allow(deprecated)] static COUNTER: AtomicUsize = ATOMIC_USIZE_INIT; let tmpdir = env::temp_dir(); for _ in 0..TRIES { let count = COUNTER.fetch_add(1, Ordering::SeqCst); let path = tmpdir.join("rust-walkdir").join(count.to_string()); if path.is_dir() { continue; } fs::create_dir_all(&path) .map_err(|e| err!("failed to create {}: {}", path.display(), e))?; return Ok(TempDir(path)); } Err(err!("failed to create temp dir after {} tries", TRIES)) } /// Return the underlying path to this temporary directory. pub fn path(&self) -> &Path { &self.0 } }